PaperlayerView exact revision on arXiv

Source-linked AI summary

DeepResearchEval: An Automated Framework for Deep Research Task Construction and Agentic Evaluation

Yibo Wang, Lei Wang, Yue Deng, Keming Wu, Yao Xiao, Huanjin Yao, Liwei Kang, Hai Ye, Yongcheng Jing, Lidong Bing

arXiv:2601.09688v1cs.CL

TL;DR

Deep research evaluation is limited by costly task construction, static quality rubrics, and factuality checks that miss uncited claims. DeepResearchEval automates persona-driven task generation and combines adaptive quality scoring with active web-based fact-checking. Across nine systems and 900 reports, it reveals substantial performance differences and systematically lower task-specific than general scores, while remaining English-centric and computationally costly.

  • Problem

    Existing benchmarks require annotation-intensive task construction, use static evaluation dimensions, and fail to verify uncited factual claims reliably.

  • Method

    DeepResearchEval uses persona-driven task generation with two-stage filtering, adaptive task-specific quality evaluation, and active external-evidence fact-checking.

  • Results

    Across 900 reports from nine systems, the framework reveals substantial performance differences and systematically lower task-specific scores than fixed general-dimension scores.

  • Takeaways & Limitations

    The framework supports fine-grained evaluation of report quality and factuality across realistic, complex deep research tasks.

  • Takeaways & Limitations

    The benchmark, evidence sources, and reporting pipelines are grounded in English-speaking information ecosystems, leaving multilingual performance unexplored.

Abstract

from arXiv · show

Deep research systems are widely used for multi-step web research, analysis, and cross-source synthesis, yet their evaluation remains challenging. Existing benchmarks often require annotation-intensive task construction, rely on static evaluation dimensions, or fail to reliably verify facts when citations are missing. To bridge these gaps, we introduce DeepResearchEval, an automated framework for deep research task construction and agentic evaluation. For task construction, we propose a persona-driven pipeline generating realistic, complex research tasks anchored in diverse user profiles, applying a two-stage filter Task Qualification and Search Necessity to retain only tasks requiring multi-source evidence integration and external retrieval. For evaluation, we propose an agentic pipeline with two components: an Adaptive Point-wise Quality Evaluation that dynamically derives task-specific evaluation dimensions, criteria, and weights conditioned on each generated task, and an Active Fact-Checking that autonomously extracts and verifies report statements via web search, even when citations are missing.

1 Introduction

Deep research systems conduct iterative web investigation and synthesis, but evaluating their long reports remains difficult because existing benchmarks are costly to construct, use static criteria, and overlook uncited claims. DeepResearchEval addresses these gaps with automated task construction and agentic evaluation, and its benchmark reveals differing strengths across systems and a gap between general and task-specific quality.

  • Deep research systems autonomously browse, retrieve, verify sources, and synthesize comprehensive citation-grounded reports.
  • Existing benchmarks rely on annotation-intensive task construction, static evaluation dimensions, and citation-linked factuality checks that leave uncited claims unexamined.
  • DeepResearchEval constructs realistic, complex tasks from personas and applies Task Qualification and Search Necessity Filters to retain 100 high-quality tasks.
  • Adaptive Point-wise Quality Evaluation adds task-specific dimensions, criteria, and weights, while Active Fact-Checking verifies cited and uncited claims with structured labels.
  • Across 900 reports from nine systems, task-specific scores are systematically lower than fixed general-dimension scores.The evaluation also reports distinct system strengths in report quality and factual evaluation.

2 Related Works

Deep research systems are specialized agents for long-horizon investigation and structured synthesis, motivating benchmarks that assess their diverse capabilities. Existing benchmark work spans multiple research-oriented evaluations, while task collection remains associated with expert-driven limitations.

  • Deep research systems autonomously plan long-horizon workflows, navigate heterogeneous web sources, and synthesize structured citation-grounded reports.
  • Existing benchmarks target a broad range of agentic capabilities, including research-style tasks such as GAIA and Humanity’s Last Exam.

3 Task Construction

The task-construction pipeline automates persona-based generation and applies qualification and search-necessity filters to produce realistic, complex research tasks. Expert validation and ranking support a 100-task benchmark intended for continued refresh.

  • Expert-driven task collection is costly, background-dependent, and static, motivating an automated alternative.
  • Construction Pipeline: The pipeline generates domain-conditioned personas, produces expertise-aligned tasks, and applies multiple quality-filtering stages.
  • Task Construction: Each persona yields four candidates requiring multi-round searches, diverse evidence integration, analytical depth, and concrete time-constrained deliverables.The generation process produces 200 candidates in total.
  • Task Filtering: The Task Qualification Filter retains tasks requiring current knowledge, multi-source integration, deep investigation, and persona alignment, using a confidence threshold above 0.7.
  • Task Filtering: The Search Necessity Filter removes tasks solvable with parametric knowledge alone, leaving 155 retained tasks before final curation.
  • Validation and Selection: 80% of retained tasks were approved by at least four experts, and 100 high-quality tasks were selected for a dynamic benchmark.The 100-task selection reflects evaluation-cost constraints rather than a pipeline deficiency.

4 Agentic Evaluation

The agentic evaluation pipeline combines task-aware quality scoring with active, statement-level fact-checking. It adapts evaluation criteria to each task and retrieves external evidence to classify both cited and uncited claims.

  • The pipeline contains an adaptive point-wise quality evaluator and an active fact checker that verifies cited and uncited statements.
  • Adaptive Point-wise Quality Evaluation: Fixed rubrics are insufficient for reports that vary across tasks, while manually customized rubrics are labor-intensive and difficult to scale.
  • Adaptive Point-wise Quality Evaluation: The evaluator combines four general dimensions with task-specific dimensions, generated criteria, and relative weights for task-aware scoring.General dimensions include Coverage, Insight, Instruction-following, and Clarity.
  • Adaptive Point-wise Quality Evaluation: The resulting score supports task-specific overall evaluation and granular analysis of dimensions and criteria.
  • Active Fact-Checking: Citation-based checking misses uncited claims and tests source support rather than factual correctness, motivating active external-evidence retrieval.
  • Active Fact-Checking: The fact checker segments reports, extracts statements, retrieves evidence, and assigns Right, Wrong, or Unknown labels using surrounding context and the task.

5 Experiments

Experiments evaluate nine deep research systems on report quality, factuality, and evaluation reliability. Results show clear quality stratification, task-specific weaknesses, factuality variation, and robust evaluation rankings.

  • Experimental Setup: Nine systems were evaluated using 100 reports per system, with quality and factuality assessed across the benchmark.The evaluation covers 900 reports in total.
  • Overall Quality Evaluation: Gemini-2.5-Pro Deep Research achieved the highest average quality score at 8.51, followed by Claude-Sonnet-4.5 at 7.53.Gemini led across all evaluated quality dimensions.
  • Overall Quality Evaluation: DeepSeek and Manus had lower overall quality scores of 5.25 and 5.95, respectively, despite moderate Instruction-following scores of 7.2 and 8.3.Their weaker Coverage and Insight scores contributed to the lower overall results.
  • Overall Quality Evaluation: Task-specific scores were consistently lower than general scores across all systems, revealing a recurring weakness in optimizing reports for task-specific criteria.The authors identify this gap as motivation for adaptive evaluation dimensions.
  • Factual Evaluation: Manus, Gemini-2.5-Pro, and DeepSeek achieved factual ratios above 76%, while statement volumes ranged from DeepSeek’s 25.08 to Gemini-2.5-Pro’s 86.99.The results suggest a potential trade-off between factual accuracy and the volume of reported statements.
  • Validation of Evaluation Methods: Evaluation rankings were robust: seven of nine model positions matched between judges, and three independent runs left rankings unchanged.GPT-5 produced lower scores than Gemini-2.5-Pro but largely preserved the ranking.
  • Validation of Evaluation Methods: The active fact-checking agent achieved 73% agreement with human experts, and automated judgments were correct in 70% of the manually re-annotated inconsistent cases.The comparison treated both Wrong and Unknown claims as negative for agreement analysis.

6 Conclusions

DeepResearchEval provides an automated framework for constructing deep research tasks and evaluating report quality and factuality. Experiments across nine systems reveal substantial performance differences.

  • Contributions: DeepResearchEval automates persona-driven task construction, adaptive report-quality evaluation, and active fact-checking through external evidence retrieval.The framework evaluates both report quality and factuality.
  • Conclusion: Experiments on nine deep research systems reveal substantial performance differences, demonstrating the framework’s effectiveness in evaluating generated long reports.

Limitations

The framework remains limited by English-centric data and by the computational and financial costs of its agentic evaluation pipeline. These constraints leave multilingual performance unexplored and limit large-scale or real-time deployment under restricted resources.

  • Language Scope: The benchmark tasks, evidence sources, and reporting pipelines are grounded in English-speaking information ecosystems, leaving multilingual performance unexplored.The ability to synthesize evidence across diverse languages also remains unexamined.
  • Scalability: Frequent frontier-model interactions and extensive search-API usage impose substantial computational and financial costs.The pipeline uses Gemini-2.5-Pro for quality scoring and GPT-5-mini for factual verification.
  • Scalability: The multi-turn, tool-intensive fact-checking design constrains scalability for large-scale or real-time deployment under limited resources.

B Deep Research Systems Details

The evaluation reports were collected during specified 2025 time windows, and average output length was measured across evaluated tasks. Most systems produced outputs exceeding 10,000 characters on average, with several generating several tens of thousands.

  • Report collection occurred during defined time windows in 2025.The collection periods are documented in Table 7.
  • Avg Length measures each system’s average valid output length across all evaluated tasks.
  • Most deep research agents exceeded 10,000 characters on average, while Gemini-2.5-Pro, Doubao, and OpenAI Deep Research reached several tens of thousands.

C More Results

Table 8 presents the full results of adaptive point-wise quality evaluation using a GPT-5 judge.

  • Table 8 reports adaptive point-wise quality evaluation results.
  • The evaluation uses GPT-5 as the judge.
  • The reported results are organized in Table 8.

D.1 Adaptive Point-wise Quality Evaluation

Adaptive Point-wise Quality Evaluation combines four general dimensions with automatically generated task-specific dimensions, criteria, and weights. Prompted evaluators assign task-grounded weights and criteria under explicit non-overlap and normalization constraints.

  • Evaluation dimensions: The framework evaluates reports using Coverage, Insight, Instruction-following, and Clarity as general dimensions.These dimensions cover coverage breadth and depth, analytical value, requirement compliance, and readability.
  • Task-specific dimensions: It automatically generates one to three additional task-specific dimensions beyond the four general dimensions.The additional dimensions are intended to capture specialized quality factors relevant to each query.
  • Criteria generation: Each dimension receives one to ten criteria scored from 0 to 10 with two-decimal precision.
  • Prompted evaluation pipeline: The evaluation prompts are designed to return JSON-formatted outputs for dimension generation, weighting, criteria generation, and scoring.The appendix presents the prompts for these stages and identifies GPT-5 as the scoring judge.
  • Weight generation: Weights are assigned dynamically to all fixed and provided task-specific dimensions, with the total constrained to exactly 1.0.The weighting prompt requires task-grounded justification and prohibits introducing new or factuality-related dimensions.
  • Criteria generation: Criteria generation requires diverse, non-overlapping, task-relevant criteria whose weights sum to 1.0 within each dimension.The prompts request explanations and weight rationales for every criterion.

E Task Construction Details

Task construction begins with domains and persona profiles, then generates persona-aligned deep research queries and filters them for genuine needs for current, multi-source investigation. The prompts specify realistic complexity, source integration, time constraints, and structured qualification outputs.

  • Persona-driven generation: The pipeline defines empirical domains, generates multiple personas per domain, and creates tasks tailored to each persona.Personas include roles, affiliations, backgrounds, and subdomains.
  • Persona-driven generation: The construction setup uses 10 fixed domains, 5 personas per domain, and 4 related deep research tasks for each persona.
  • Search Necessity Filter: Search Necessity generates a no-search baseline and discards tasks that the baseline can already answer well.The baseline is produced solely from existing knowledge without external search tools and evaluated for accuracy, completeness, depth, timeliness, professionalism, and structure.
  • Query requirements: Each query must require at least two web-search rounds from different perspectives and integrate multiple credible source types.Examples include academic papers, industry reports, news articles, policy documents, statistics, and online forums.
  • Query requirements: Queries are required to match persona complexity, provide concrete deliverables, and generally address current or emerging topics with time-sensitive evidence.The specification calls for 70% of queries to include explicit time constraints.
  • Task Qualification Filter: Task Qualification checks whether queries need current information, cross-verification, multiple credible sources, and multi-angle investigation.The filter also evaluates whether complexity matches the persona’s role and capabilities.

F.1 Correct Examples Case 1

The examples show active fact-checking distinguishing incorrect claims caused by numerical or evidential errors from claims that remain insufficiently verified. They also illustrate occasional over-strict judgments that incorrectly reject supported statements.

  • DeepSeek V3 compute claim: 2,788M incorrectly denotes 2,788 million GPU hours, whereas the cited figure is 2.788M, creating a three-orders-of-magnitude unit error.The associated $5.6M figure is a community-derived estimate based on assumed GPU hourly costs, not an explicitly reported training cost.
  • FECC claim: The FECC claim is classified as Unknown because its duration varies across sources and the precise 15–20 player age range is not explicitly substantiated.The program’s prestige and focus on developing young-player coaches are broadly supported.
  • Semiconductor export-controls claim: The EU semiconductor-controls claim is Unknown because evidence supports Dutch expansion for immersion DUV tools but not EU-wide controls adding EUV pellicles.The statement therefore overstates the scope of the documented regulatory action.
  • Scope of the issue: Overly strict verification produces incorrect judgments only occasionally, despite the two examples presented.This caveat limits the reported scope of the observed verification problem.
  • Overly strict verification: The model incorrectly rejected claims about AI triage effectiveness and basic-system procurement by applying an overly strict requirement for universal or authoritative support.The final judgments state that concrete examples and the explicitly reported $50,000–$200,000 range were sufficient.

G Examples

The examples demonstrate task-specific evaluation dimensions and factual verification judgments across comparative research tasks. They show how the framework assesses synthesis quality while distinguishing unsupported precision, mixed-accuracy claims, and incorrect numerical statements.

  • Transportation task: Task-specific dimensions include safety metrics, policy pragmatism, comparative synthesis, and metric utility for cross-jurisdictional transportation research.These dimensions assess measurable and actionable metrics, implementation across distinct contexts, integrated regulatory analysis, and coherent synthesis rather than simple juxtaposition.
  • Health task: The health example evaluates classification rigor, cross-regional synthesis, and recommendation actionability for plant-based meat products in the US and EU.The dimensions address scientific application of the NOVA framework, regional formulation and regulatory differences, and evidence-based public-health recommendations.
  • Sports task: The sports example assesses implementation viability and comparative synthesis for youth basketball systems, including scalable municipal guidance and transferable lessons.Implementation viability covers actionable templates, realistic budgets, and adaptation to city size, funding, and infrastructure.
  • Factual evaluation examples: China’s consumer-electronics share is judged Wrong because authoritative estimates place it around 40–45%, not two-thirds, despite its status as the largest manufacturing hub.The evidence supports China’s dominant manufacturing role but not the statement’s approximately 66% figure.
  • Factual evaluation examples: The SMIC claim is Wrong because 7nm production without EUV is supported, while stable 5nm mass production remained unconfirmed and speculative through mid-2024.The mixed statement receives an overall negative judgment under the stated verification rule.
  • Factual evaluation examples: The 27% claim about Americans’ views of AI is Wrong because Pew findings report different figures, including 44% for positive healthcare impact and 17% for positive US impact.The cited reports also describe a substantial gap between expert and public views.
Loading 2601.09688v1…