PaperlayerView exact revision on arXiv

Source-linked AI summary

Understanding DeepResearch via Reports

Tianyu Fan, Xinyao Niu, Yuxiang Zheng, Fengji Zhang, Chengen Huang, Bei Chen, Junyang Lin, Chao Huang

arXiv:2510.07861v1cs.AI

TL;DR

DeepResearch evaluation lacks a holistic way to assess integrated research outputs. DEEPRESEARCH-REPORTEVAL evaluates reports across three dimensions, and its expert-aligned assessment of four commercial systems reveals distinct design philosophies and performance trade-offs.

  • Problem

    Existing benchmarks assess isolated capabilities, while DeepResearch reports require evaluation of integrated synthesis, coherent insights, and evidence grounding.

    Source: §1 · p. 1

  • Method

    DEEPRESEARCH-REPORTEVAL uses LLM-as-a-Judge with expert judgment to assess report quality, redundancy, and factuality.

    Source: §1 · p. 2

  • Results

    Four commercial systems show distinct design philosophies and performance trade-offs; the framework achieves 61.11% ranking agreement with human experts.

    Source: §2.3.4 · p. 7, §6 · p. 10

  • Takeaways & Limitations

    The benchmark provides 100 representative queries across 12 categories for evaluating current research automation capabilities.

    Source: §6 · p. 10

    Report-based end-to-end evaluation may obscure intermediate processes and system behaviors that merit independent assessment.

    Source: §4.3 · p. 9

Abstract

from arXiv · show

DeepResearch agents represent a transformative AI paradigm, conducting expert-level research through sophisticated reasoning and multi-tool integration. However, evaluating these systems remains critically challenging due to open-ended research scenarios and existing benchmarks that focus on isolated capabilities rather than holistic performance. Unlike traditional LLM tasks, DeepResearch systems must synthesize diverse sources, generate insights, and present coherent findings, which are capabilities that resist simple verification. To address this gap, we introduce DeepResearch-ReportEval, a comprehensive framework designed to assess DeepResearch systems through their most representative outputs: research reports. Our approach systematically measures three dimensions: quality, redundancy, and factuality, using an innovative LLM-as-a-Judge methodology achieving strong expert concordance. We contribute a standardized benchmark of 100 curated queries spanning 12 real-world categories, enabling systematic capability comparison. Our evaluation of four leading commercial systems reveals distinct design philosophies and performance trade-offs, establishing foundational insights as DeepResearch evolves from information assistants toward intelligent research partners. Source code and data are available at: https://github.com/HKUDS/DeepResearch-Eval.

1 INTRODUCTION

DeepResearch agents clarify research intent, gather and validate evidence, and synthesize findings, but existing benchmarks assess isolated capabilities rather than integrated research performance. This paper proposes evaluating these systems through their research reports.

1 INTRODUCTION

  • Existing benchmarks emphasize search strategies or multi-hop reasoning, missing the holistic demands of end-to-end research reports.The paper identifies a need to assess how systems turn queries into research objectives, synthesize diverse sources, and produce coherent, evidence-grounded insights.Source: §1 · p. 1
  • DeepResearch systems clarify user intent, investigate sources with reasoning and tools, then synthesize findings into structured, traceable reports.The report may include charts, tables, and images; alternative outputs can include webpages and podcasts.Source: §1 · pp. 1–2
  • DEEPRESEARCH-REPORTEVAL combines LLM-as-a-Judge scoring with expert judgment to assess report quality, redundancy, and factuality.The framework uses expert judgment to support the reliability of its assessments.Source: §1 · p. 2
  • The authors release 100 queries and corresponding Qwen DeepResearch reports to facilitate systematic evaluation.The benchmark queries span diverse categories.Source: §1 · p. 2

2 THE FRAMEWORK OF DEEPRESEARCH-REPORTEVAL

Because DeepResearch queries demand broad synthesis and nuanced insights, this section presents reports as an evaluation medium and introduces a framework for scoring report quality, redundancy, and factuality with expert alignment.

2.1 WHY EVALUATE DEEPRESEARCH AGENTS THROUGH REPORT QUALITY?

  • Across more than 150K Qwen queries, twelve categories emerged, led by Science & Technology (37.3%) and Economy & Business (17.2%).The authors report that these domains require systematic research and decisions based on comprehensive understanding rather than isolated facts.Source: §2.1 · p. 3

2.2 WHAT MAKES DEEPRESEARCH REPORT EVALUATION CHALLENGING?

  • Traditional writing metrics miss scope-sensitive coverage, source reliability, citation accuracy, and analytical insight; ReportEval directly assesses individual reports rather than matching reference reports.DeepResearch reports must balance factual evidence with meaningful interpretation, not merely enumerate facts.Source: §2.2 · pp. 3–4

2.3 HOW DO WE EVALUATE DEEPRESEARCH REPORTS?

  • ReportEval assesses report quality, redundancy, and factuality using LLM judges and human experts; its current scope is report text and tables, not images or charts.Human experts are involved to help align scores with expert judgments.Source: §2.3 · p. 4

2.3.1 QUALITY

  • Quality evaluation covers comprehensiveness, coherence, clarity, insightfulness, and overall quality, with LLM judges assigning scores from 0 to 4 and providing justifications.The dimensions assess coverage, organization, language, analytical viewpoints, and holistic effectiveness.Source: §2.3.1 · pp. 4–5

2.3.2 REDUNDANCY

  • Redundancy evaluation segments reports into paragraphs, has an LLM judge all paragraph pairs for semantic independence, and averages pair scores, where 4 means no redundancy.The pairwise procedure targets repeated viewpoints, examples, or expressions across sections.Source: §2.3.2 · p. 5

2.3.3 FACTUALITY

  • Factuality evaluation checks each claim against each cited source, then reports mean support and full-support rate to measure claim-source alignment.A claim citing multiple sources is evaluated independently against each source.Source: §2.3.3 · pp. 5–6

2.3.4 LLM-HUMAN ALIGNMENT PROCESS

  • After expert-guided iterative prompt tuning, final scores had consistently small MADs, while human and model rankings matched exactly in 61.11% of cases.The ranking comparison used reports generated by different LLMs for 120 queries.Source: §2.3.4 · pp. 6–7

3 EXPERIMENTAL EVALUATION

The evaluation uses 100 real-world-oriented queries to compare four commercial DeepResearch systems. Results show differing trade-offs across analytical quality, report length, and evidence grounding.

Experimental Setup

  • The benchmark contains 100 categorized queries whose distribution closely reflects real-world user behavior and ranges from specific requirements to open-ended inquiries.The four evaluated systems are OpenAI, Perplexity, Gemini, and Qwen; GPT-4o judges English queries and reports.Source: §3 · p. 7

Evaluation Metrics and Statistics

  • The evaluation reports length in tokens, assesses redundancy using paragraph pairs, and measures factuality through claim–source pairs.Redundancy pairing is capped at 30 because of quadratic scaling, and introductory and concluding paragraphs are excluded.Source: §3 · p. 8

Length vs. Quality Trade-offs

  • 1245.2 tokens is Perplexity’s average report length; it scores highly in coherence and clarity and has low redundancy at 3.71, but its comprehensiveness and insightfulness scores are lowest.The authors describe a conciseness and readability priority that trades off with analytical depth.Source: §3 · p. 8

Finding the Sweet Spot in Report Length

  • OpenAI and Qwen reports average approximately 5000–7000 tokens, while Gemini averages over 9000; quality scores are generally similar, with Qwen slightly higher.The authors report that report length does not have a linear relationship with quality.Source: §3 · p. 8

Analytical Excellence: When Depth Meets Readability

  • Qwen achieves the best performance in comprehensiveness and insightfulness while maintaining readability.The authors attribute this to analytical depth in its reports.Source: §3 · p. 8

Evidence-Based Credibility: The Foundation of Trust

  • Qwen and OpenAI lead on factuality, with high average support scores and strong support rates alongside the largest average numbers of claim–source pairs.The authors interpret this as evidence that cited sources substantiate most report claims.Source: §3 · p. 8

4 REFLECTIONS ON DEEPRESEARCH

The reflections examine query refinement and research-oriented search, the limits of report-level evaluation, and possible directions for expanding DeepResearch capabilities. They connect user interaction and evidence gathering with broader questions about assessment and system design.

4.1 THE ART OF QUERY REFINEMENT

  • Clarification-question quality positively correlates with final-report quality, suggesting the initial interaction is a critical foundation across the research pipeline.Source: §4.1 · p. 8
  • Proactively asking about knowledge gaps, conflicting views, and unaddressed aspects could reduce uncertainty and steer research toward deeper insights.Source: §4.1 · pp. 8–9

4.2 RETHINKING SEARCH FOR RESEARCH

  • Search agents strong on answer-centric benchmarks may not meet DeepResearch’s need to gather diverse supporting evidence for complex questions.Source: §4.2 · p. 9
  • Research-oriented retrieval benchmarks could measure evidence coverage, perspective diversity, cross-source consistency, and opposing viewpoints, while training encourages recall of diverse relevant fragments.Source: §4.2 · p. 9

4.3 EXPANDING THE EVALUATION LANDSCAPE

  • End-to-end report evaluation may obscure intermediate processes and system behaviors that merit independent assessment.Source: §4.3 · p. 9
  • Search depth and end-to-end speed can trade off, while parallel tool use or sub-agent collaboration may improve efficiency.Source: §4.3 · p. 9

4.4 ENVISIONING THE DEEPRESEARCH AGENT

  • Customized tools such as private-database interfaces could provide access to specialized repositories and proprietary datasets beyond web-based research.Source: §4.4 · p. 9
  • Confidence scores or dispute indicators could help users judge reliability, while proactive content delivery could shift the system from reactive tool to research partner.Source: §4.4 · p. 9

5 RELATED WORK

Related work covers DeepResearch systems, knowledge-enhanced agents, and evaluation benchmarks. It describes progress in research automation alongside limitations in assessing comprehensive report outputs.

6 CONCLUSION

The paper presents DEEPRESEARCH-REPORTEVAL for assessing DeepResearch through research reports. Its evaluation finds expert alignment and distinct performance trade-offs among commercial systems.

  • DEEPRESEARCH-REPORTEVAL evaluates research reports across quality, redundancy, and factuality as representative outputs of DeepResearch systems.Source: §6 · p. 10
  • Low MAD values and 61.11% ranking agreement demonstrate strong concordance with expert judgments after iterative LLM-human alignment.Source: §6 · p. 10
  • The benchmark contains 100 representative queries across 12 categories, and evaluation of four leading commercial systems reveals distinct design philosophies and performance trade-offs.Source: §6 · p. 10

A EXAMPLES FROM DEEPRESEARCH-REPORTEVAL

Appendix Table A1 provides representative DeepResearch queries from the framework’s categories. The examples are described as complex and open-ended.

B PROMPTS FOR LLM-AS-A-JUDGE EVALUATION

The appendix specifies LLM-as-a-Judge prompts and scoring rubrics for report quality, redundancy, and factuality. The rubrics distinguish content-level repetition from superficial similarity and assess factual claims against supplied web content.

B.1 QUALITY

  • Quality scores assess comprehensiveness and depth, structural clarity and logic, fluency and consistency, and material integration and originality.The prompt asks evaluators to score each criterion out of four.Source: §B.1 · p. 15

B.2 REDUNDANCY

  • Redundancy means repeated viewpoints, content, examples, data, or references, including when the same information is paraphrased.The prompt distinguishes explicit repetition from implicit repetition based on equivalent information rather than wording.Source: §B.2 · p. 17
  • Redundancy assessment considers explicit copying and implicit restatement, while weighing effects on readability and information density without judging factual correctness.The prompt also instructs evaluators to avoid subjective bias and not rely on external knowledge.Source: §B.2 · p. 17
  • The redundancy rubric ranges from 4 for independent paragraphs to 0 when almost all content is repeated and the writing loses value.Intermediate scores distinguish slight repetition from multiple instances, severe repetition, and their effects on reading or writing quality.Source: §B.2 · pp. 17–18
  • Related topics, different wording alone, and supplementary material are not repetition unless the underlying information or viewpoint is repeated.Using the same example to explain different concepts is also excluded from repetition.Source: §B.2 · pp. 17–18

B.3 FACTUALITY

  • Factuality is judged by whether each report sentence is directly supported by, or reasonably inferable from, the supplied web content.The prompt directs evaluators to check key terms such as time, location, people, and quantities, including every information point in a multi-claim sentence.Source: §B.3 · pp. 21–22
  • Factuality outputs a support label and supporting web-content sentences, distinguishing fully supported, partially supported, and unsupported claims.The prompt's output labels are 1, 0, and -1, respectively; unsupported includes claims contradicted by the web content.Source: §B.3 · p. 22

Related papers

Loading 2510.07861v1…