Source-linked AI summary
ReportBench: Evaluating Deep Research Agents via Academic Survey Tasks
Minghao Li, Ying Zeng, Zhihao Cheng, Cong Ma, Kai Jia
TL;DR
Research reports from Deep Research agents demand factual accuracy and comprehensive coverage, yet mature evaluation methods are lacking. ReportBench builds survey-grounded tasks and validates report references and statements; experiments show Deep Research products can outperform base models.
Problem
Research reports require factual accuracy and comprehensive content, but evaluation lacks mature methodologies and broad consensus on what makes a good report.
Source: §1 · pp. 1–2Method
ReportBench derives prompts and reference sets from expert-authored arXiv surveys and evaluates reference quality alongside cited- and non-cited-statement factuality.
Source: §1 · p. 2Results
Deep Research products can outperform base models in the authors’ large-scale experiments.
Source: §6 · p. 10Takeaways & Limitations
ReportBench is intended to help researchers monitor, compare, and improve AI-system reliability on academic survey tasks.
Source: §6 · p. 11To mitigate legal risk, the dataset includes only papers under specified permissive or arXiv distribution licenses and contains essential metadata only.
Source: §A.1 · p. 15
Abstract
from arXiv · showhide
The advent of Deep Research agents has substantially reduced the time required for conducting extensive research tasks. However, these tasks inherently demand rigorous standards of factual accuracy and comprehensiveness, necessitating thorough evaluation before widespread adoption. In this paper, we propose ReportBench, a systematic benchmark designed to evaluate the content quality of research reports generated by large language models (LLMs). Our evaluation focuses on two critical dimensions: (1) the quality and relevance of cited literature, and (2) the faithfulness and veracity of the statements within the generated reports. ReportBench leverages high-quality published survey papers available on arXiv as gold-standard references, from which we apply reverse prompt engineering to derive domain-specific prompts and establish a comprehensive evaluation corpus. Furthermore, we develop an agent-based automated framework within ReportBench that systematically analyzes generated reports by extracting citations and statements, checking the faithfulness of cited content against original sources, and validating non-cited claims using web-based resources. Empirical evaluations demonstrate that commercial Deep Research agents such as those developed by OpenAI and Google consistently generate more comprehensive and reliable reports than standalone LLMs augmented with search or browsing tools. However, there remains substantial room for improvement in terms of the breadth and depth of research coverage, as well as factual consistency. The complete code and data will be released at the following link: https://github.com/ByteDance-BandAI/ReportBench
1 Introduction
Research reports from Deep Research agents require rigorous content evaluation, but mature evaluation methods are lacking. ReportBench uses expert-authored surveys as references and evaluates report references and statement factuality.
1 Introduction
- Practical adoption of Deep Research agents depends on research reports being factually accurate and comprehensive.These agents can complete tasks that previously required days or weeks in minutes.Source: §1 · p. 1
- Research-report evaluation lacks mature methodologies, in part because there is no broad consensus on what constitutes a good report.The paper focuses on content quality, leaving subjective writing-style evaluation to future work.Source: §1 · p. 2
- ReportBench uses published arXiv survey papers as gold-standard references, drawing on expert authorship and peer review.The authors describe this as a way to address variation in human annotator expertise.Source: §1 · p. 2
- ReportBench evaluates report content through reference quality and relevance, and the faithfulness and veracity of generated statements.For cited statements, it checks support in cited sources; for non-cited statements, it uses web-connected models for factual verification.Source: §1 · p. 2
- The authors release datasets, prompts, and evaluation scripts to support reproducible research and community-driven progress.Source: §1 · p. 2
2 Methodology
ReportBench constructs survey-based research tasks and evaluates generated reports through reference comparison and separate checks of cited and non-cited statements. Its dataset pipeline identifies published surveys, generates temporally bounded prompts, classifies application domains, and selects a balanced set of tasks.
2 Methodology
- ReportBench’s methodology combines benchmark construction from expert-authored surveys with a validation pipeline for report quality and factual consistency.Source: §2 · p. 3
2.1 Dataset Construction
- The dataset pipeline identifies survey papers, extracts their references, generates prompts, and assigns application domains.Figure 1 presents the overall data construction workflow.Source: §2.1 · p. 3
2.1.1 Phase I: Survey Paper Identification and Reference Extraction
- Survey identification filters for post-2020 papers, evidence of peer review and publication, survey- or review-related titles, and survey classification using GPT-4o.The process begins with the arXiv metadata snapshot.Source: §2.1.1 · p. 3
2.1.2 Phase II: Prompt Generation
- Prompts are reverse-engineered from each survey’s publication date and full text, with sentence-, paragraph-, and detail-rich versions and a cutoff date matching the survey’s update horizon.The prompts also instruct models not to cite the original survey after the authors observed models disregarding temporal constraints and retrieving it directly.Source: §2.1.2 · pp. 3–4
2.1.3 Phase III: Application Domain Distribution
- Because the arXiv corpus creates domain imbalance, the authors down-sample 100 papers, randomly select among their prompt types, and validate subset classification with four experts.Figure 2 shows the domain proportions and task counts across 11 categories among 678 filtered prompts.Source: §2.1.3 · p. 4, Figure 2 · p. 5
2.2 Evaluation Process
- Reports are generated under a time limit and a restriction against referencing the original survey, then assessed through reference overlap and statement validation.Cited statements are checked against retrieved source passages; non-cited factual statements are checked independently by multiple web-connected models and judged by voting.Source: §2.2 · pp. 4–6
3 Experiment
The experiment evaluates Deep Research agents and search-augmented base models using reference and statement-quality metrics. Deep Research products outperform their base models overall, while the product comparison shows trade-offs in retrieval and statement quality.
3 Experiment / 3.1 Setttings
- The benchmark compares native OpenAI and Gemini Deep Research agents with base models augmented by search and link-reading tools.Responses were collected from web interfaces and batched base-model runs during July 14–25.Source: §3 · p. 6, §3.1 · p. 6
3.2 Evaluation Metrics
- The evaluation measures reference precision, recall, and density, plus statement counts, cited-statement match rate, and non-cited factual accuracy.Cited-statement match rate measures semantic consistency with cited sources; non-cited accuracy is verified through web-connected LLMs.Source: §3.2 · p. 7, Table 1 · p. 7
3.3 Product-Level Comparative Analysis
- OpenAI Deep Research has higher reference precision than Gemini (0.385 vs. 0.145), while Gemini’s recall is slightly higher (0.036 vs. 0.033).Gemini averages 32.42 cited statements per report versus OpenAI’s 9.89, without a significant recall improvement.Source: §3.3 · p. 7, Table 1 · p. 7
- OpenAI’s citation match rate (78.87% vs. 72.94%) and non-cited statement accuracy (95.83% vs. 92.21%) exceed Gemini’s.Gemini produces more non-cited statements on average (49.6 vs. 38.9).Source: §3.3 · p. 7, Table 1 · p. 7
3.4 Model-Level Comparative Analysis
- Against their base models, both Deep Research products generate more statements; OpenAI also has higher citation match and non-cited factual accuracy than o3, while Gemini trades precision for recall relative to gemini-2.5-pro.OpenAI Deep Research versus o3: match rate 78.87% vs. 31.43% and factual accuracy 95.83% vs. 82.22%; Gemini Deep Research versus gemini-2.5-pro: precision 0.145 vs. 0.269 and recall 0.036 vs. 0.010.Source: §3.4 · p. 8
- Deep Research products significantly outperform their base models in coverage and factual grounding, though hallucinations and over-citation remain challenges.The authors point to task-specific fine-tuning or pipeline design beyond standalone LLM capabilities as possible contributors.Source: §3.4 · p. 8
4 Analysis
The analysis identifies author misattribution and fabricated links as citation-related errors in generated reports. The authors argue that citation semantic consistency is sensitive to such discrepancies and can capture them in evaluation.
Hallucination patterns
- The analysis identifies statement hallucination and citation hallucination, including content that deviates from a cited source and fabricated references.The authors say citation semantic consistency can quantitatively capture and penalize these discrepancies.Source: §4 · pp. 8–9
Statement Hallucination
- In one example, a report attributed a reasoning-enhanced Text-to-SQL RL reward scheme to Kulkarni et al., although Kulkarni was not among the cited paper’s authors.The authors speculate that similar training data may have led to the mistaken attribution.Source: §4 · p. 9
Citation Hallucination
- In another example, Gemini generated a crowd-monitoring claim with a cited URL that the authors found did not exist and appeared fabricated.The authors describe fabricated plausible-looking links as an error mode in function-call–driven retrieval.Source: §4 · p. 9
5 Related Work
Prior work addresses fact checking, citation evaluation, survey generation, and Deep Research capabilities, but the paper distinguishes ReportBench by its focus on automatically verifiable report content and citation completeness.
Fact Checking Evaluation
- Fact-checking benchmarks evaluate components such as claim detection and evidence retrieval, rather than synthesized information as a whole or full research reports.The paper frames long-form LLM reports as a limitation of component-focused evaluation.Source: §5 · pp. 9–10
Citation Evaluation
- Citation evaluation has studied citation correctness and existence, while completeness of related-work coverage remains underexplored.ReportBench uses LLMs and search tools to assess completeness alongside other report-quality dimensions.Source: §5 · p. 10
Survey Generation
- Survey-generation work ranges from hierarchical outlines and citation-network methods to SurveyForge’s full survey generation; ReportBench instead focuses on factual faithfulness and correctness.The paper also contrasts its automated construction pipeline with SurveyBench’s broader evaluation dimensions.Source: §5 · p. 10
Deep Research Evaluation
- Existing Deep Research benchmarks cover retrieval, multi-hop factual reasoning, or end-to-end report generation, which the paper says may not assess core competencies for rigorous research.ReportBench targets factual accuracy and citation behavior.Source: §5 · p. 10
6 Conclusion
The paper presents ReportBench to evaluate references and statement-level factual accuracy in Deep Research reports. Its experiments find that Deep Research products can outperform base models in coverage and factual grounding, while hallucination and over-citation remain challenges.
6 Conclusion
- ReportBench evaluates reference quality and the factual accuracy of statements in Deep Research reports using expert-authored surveys as ground truth and reverse prompt engineering.Its validation separately checks cited statements against sources and non-cited statements through web-based factual verification.Source: §6 · p. 10
- The experiments show Deep Research products can outperform base models in coverage and factual grounding, but still face hallucination and over-citation challenges.The authors hope the benchmark will help the research community monitor, compare, and improve reliability on academic survey tasks.Source: §6 · pp. 10–11
A.1 Limitations
ReportBench’s coverage and included materials are constrained by its source distribution and copyright requirements.
Data Distribution
- Most source surveys are concentrated in STEM, which may limit evaluation applicability to other research areas.Future iterations are planned to add source domains to improve coverage and generalization.Source: §A.1 · p. 15
Copyright Constraints
- Only papers under specified permissive licenses are included, and the released dataset contains essential metadata rather than full papers.The dataset is released under CC0 1.0; included paper licenses are CC BY 4.0, CC BY-SA 4.0, CC0 1.0, and the arXiv.org Non-exclusive license to distribute.Source: §A.1 · p. 15
- Narrowing the license scope further would compromise domain balance.Source: §A.1 · p. 15
A.2 Prompts in Evaluation
The appendix prompts extract cited and uncited claims, assess support and factuality, and collect academic-paper reference information.
A.2.1 Cited Statement Extraction
- Cited-statement extraction returns each externally cited claim paired with its canonical source URL, duplicating claims when citations contain multiple URLs.Source: §A.2.1 · p. 15
A.2.2 Non-cited Statement Extraction
- Non-cited-statement extraction selects specific, externally verifiable factual claims absent from the cited list and excludes opinions, cited claims, common knowledge, and vague statements.The output is a JSON list of uncited statements.Source: §A.2.2 · p. 15
A.2.3 Supporting Source Extraction
- Supporting-source extraction returns source-document content relevant to a given statement, or “NOT_FOUND” when no content supports it.Source: §A.2.3 · p. 16
A.2.4 Semantic Consistency Verification
- Semantic consistency verification decides whether a source sentence faithfully supports a report claim and returns a reason with a true-or-false match.Source: §A.2.4 · p. 16
A.2.5 Web-based Statement Verification
- Web-based verification uses reliable sources from multiple searches to judge whether a factual statement is correct or incorrect.The response includes a reason describing the verification process and findings, plus a boolean decision.Source: §A.2.5 · p. 16
A.2.6 Reference Title Extraction
- Reference-title extraction returns explicitly mentioned academic papers with titles, authors, and an academic-paper flag, excluding books, websites, and other references.Source: §A.2.6 · p. 17
A.3 Prompt in Data Construction
The data-construction prompts illustrate sentence-level, paragraph-level, and detail-rich research tasks with topical, temporal, and output constraints.
Sentence-level prompt
- A sentence-level radar prompt asks about autonomous-driving data representations and restricts references to papers published before April 2025.Source: §A.3 · p. 18
Paragraph-level prompt
- A paragraph-level LiDAR prompt requests analysis of point-cloud registration, 3D features, and deep-learning methods, with literature before November 2020 and a named-survey exclusion.Source: §A.3 · p. 18
Detail-rich prompt
- The detail-rich GNN text-classification prompt specifies corpus- and document-level approaches, representative models, datasets, metrics, challenges, pre-July 2024 literature, and top venues.Source: §A.3 · p. 18
Sentence-level prompt
- Other sentence-level prompt fragments constrain NLP and AI references by venue, exclude named surveys, and request an English survey with citations.Source: §A.3 · p. 18