PaperlayerView exact revision on arXiv

Source-linked AI summary

DeepScholar-Bench: A Live Benchmark and Automated Evaluation for Generative Research Synthesis

Liana Patel, Negar Arabzadeh, Harshit Gupta, Ankita Sundar, Ion Stoica, Matei Zaharia, Carlos Guestrin

arXiv:2508.20033v2cs.CLcs.AI

TL;DR

Generative research synthesis systems need benchmarks that reflect long-form, evolving research tasks rather than short-answer question answering, but existing alternatives are limited by staleness and contamination. DeepScholar-bench provides a live benchmark and holistic evaluation for related-work generation, and its systematic comparison finds that no evaluated system exceeds a 31% geometric mean across metrics.

  • Problem

    Existing benchmarks do not adequately evaluate realistic long-form research synthesis across retrieval, knowledge synthesis, and verifiability, while expert-curated datasets risk becoming stale.

  • Method

    DeepScholar-bench uses recent high-quality ArXiv papers and human-written exemplars to evaluate related-work generation with automated metrics spanning knowledge synthesis, retrieval quality, and verifiability.

  • Results

    No evaluated system surpasses a geometric mean of 31% across all metrics, indicating substantial opportunity for improvement.

  • Takeaways & Limitations

    DeepScholar-bench and DeepScholar-ref provide a live benchmark and reference pipeline for advancing generative research synthesis systems.

  • Takeaways & Limitations

    Reference Coverage is conservative because LLM labeling under-identifies essential references, with a 24.2% off-diagonal mass in the human evaluation.

Abstract

from arXiv · show

The ability to research and synthesize knowledge is central to human expertise and progress. A new class of AI systems--designed for generative research synthesis--aims to automate this process by retrieving information from the live web and producing long-form, cited reports. Yet, evaluating such systems remains an open challenge: existing question-answering benchmarks focus on short, factual answers, while expert-curated datasets risk staleness and data contamination. Neither captures the complexity and evolving nature of real research synthesis tasks. We introduce DeepScholar-bench, a live benchmark and automated evaluation framework for generative research synthesis. DeepScholar-bench draws queries and human-written exemplars from recent, high-quality ArXiv papers and evaluates a real synthesis task: generating a related work section by retrieving, synthesizing, and citing prior work. Our automated framework holistically measures performance across three key dimensions--knowledge synthesis, retrieval quality, and verifiability. To further future work, we also contribute DeepScholar-ref, a simple, open-source reference pipeline, which is implemented on the LOTUS framework and provides a strong baseline. Using DeepScholar-bench, we systematically evaluate prior open-source systems, search agents with strong models, OpenAI's DeepResearch, and DeepScholar-ref. We find DeepScholar-bench is far from saturated: no system surpasses a geometric mean of $31\%$ across all metrics. These results highlight both the difficulty and importance of DeepScholar-bench as a foundation for advancing AI systems capable of generative research synthesis. We make our benchmark code and data available at https://github.com/guestrin-lab/deepscholar-bench.

1 Introduction

DeepScholar-bench addresses the lack of realistic, current benchmarks for generative research synthesis by evaluating related-work generation across retrieval, knowledge synthesis, and verifiability. Its systematic evaluation finds substantial room for improvement, with no system exceeding a 31% geometric mean across metrics.

  • Motivation: Existing question-answering benchmarks emphasize short, easily verifiable answers, while expert-curated open-ended benchmarks can become stale and outdated.Neither benchmark type captures the complexity of long-form answers synthesized from many sources.
  • Benchmark design: DeepScholar-bench evaluates related-work generation through retrieval, synthesis, and citation of prior research across knowledge synthesis, retrieval quality, and verifiability.The framework uses human-written related-work sections and automated metrics designed to agree with human judgments.
  • Benchmark design: DeepScholar-bench is a live benchmark that draws queries from recent, high-quality ArXiv papers and plans monthly dataset and leaderboard updates.An automated pipeline curates the benchmark from recent papers and supports new dataset instantiations.
  • Contributions: DeepScholar-ref is a simple open-source reference pipeline, while the benchmark systematically compares 14 existing baselines and reports their metric ranges.The pipeline is implemented on LOTUS and is intended to promote future research.
  • Evaluation results: No system surpasses a geometric mean of 31% across all metrics, demonstrating that DeepScholar-bench remains difficult and unsaturated.The benchmark evaluates existing open-source systems, search agents, OpenAI’s DeepResearch, and DeepScholar-ref.

2 The DeepScholar Dataset

The DeepScholar dataset targets related-work generation using recent, conference-accepted ArXiv papers across diverse research domains. Its automated pipeline filters papers and extracts metadata, related-work sections, and cited references while controlling recency and contamination.

  • Task formulation: The dataset task is to retrieve relevant sources and generate a related-work section from a paper’s description, using human-written exemplars for evaluation.Queries are sourced from ArXiv papers accepted at academic conferences.
  • Design goals: The pipeline prioritizes diverse topics, recent papers, and peer-reviewed conference-accepted manuscripts.Recency supports timely queries and helps prevent contamination from models trained on earlier web snapshots.
  • Data pipeline: The collection pipeline scrapes, filters, and extracts paper metadata, related-work sections, and reference lists to construct dataset instances.The pipeline loads papers from configured ArXiv domains and publication-date ranges while addressing duplicate ArXiv versions.
  • DeepScholar-June-2025: DeepScholar-June-2025 covers April–June 2025 papers from 18 ArXiv domains and selects papers marked as accepted at a conference.The instantiation follows the April 5, 2025 release of Llama-4 models and applies additional filtering to related-work sections.

3 The DeepScholar Evaluation Framework

The DeepScholar evaluation framework assesses research synthesis across knowledge synthesis, retrieval quality, and verifiability using automated metrics grounded in human-written exemplars.

  • 3 The DeepScholar Evaluation Framework: Seven fine-grained metrics evaluate knowledge synthesis, retrieval quality, and verifiability to address the complexity of research synthesis.Knowledge synthesis assesses report organization, coherence, and nugget coverage; retrieval quality evaluates retrieved references; verifiability evaluates citation support and claim coverage.
  • 3 The DeepScholar Evaluation Framework: Knowledge synthesis evaluates organization, coherence, and nugget coverage against human-written exemplars.Organization and coherence use pairwise LLM judging, while nugget coverage measures essential information included in the generated report.
  • 3 The DeepScholar Evaluation Framework: Reference coverage measures the fraction of important exemplar references retrieved by a system.Important references are those considered necessary for situating the contribution, while replaceable or omissible references are labeled not-important.
  • 3 The DeepScholar Evaluation Framework: Document Importance compares the median citation count of retrieved sources with that of the exemplar reference set, capped at 1.The source-level citation count is defined by num-cites(s).
  • 3 The DeepScholar Evaluation Framework: Verifiability uses citation precision and claim coverage based on LLM-based entailment evaluations.Citation precision checks whether cited sources support claims in accompanying sentences.

4 DeepScholar-ref

DeepScholar-ref iteratively searches the web and applies semantic operators to filter, rank, and aggregate sources into a final research-synthesis report.

  • 4 DeepScholar-ref: DeepScholar-ref iteratively generates web-search queries, summarizes results, and produces new queries before final synthesis.The pipeline uses semantic filtering to remove irrelevant documents and semantic top-k ranking to prioritize relevant sources.

5 Experimental Results

DeepScholar-bench reveals substantial gaps across knowledge synthesis, retrieval quality, and verifiability, while DeepScholar-ref provides a competitive baseline. Automated judges show promising agreement with human evaluators, but reference-coverage scores remain conservative.

  • 5.1 Generative Research Synthesis Systems Demonstrate Large Room for Improvement.: No system surpasses a 31% geometric mean across all metrics, and Nugget Coverage, Reference Coverage, and Document Importance remain below 40% for every baseline.OpenAI DeepResearch obtains the highest geometric mean, underscoring the difficulty of the benchmark’s live-web, long-form synthesis task.
  • 5.1.1 Generative Research Synthesis Systems Demonstrate Large Room for Improvement.: OpenAI DeepResearch leads Organization at .857 and Nugget Coverage at .392, but systems still struggle to surface key facts despite producing coherent summaries.The results separate report organization from factual coverage: strong structure does not guarantee comprehensive research synthesis.
  • 5.1.2 Retrieval Quality: OpenAI DeepResearch reaches .629 Relevance Rate but only .187 Reference Coverage and .124 Document Importance, showing relevant retrieval without comprehensive coverage of notable sources.Its retrieval quality is therefore strongest for source relevance, while coverage and importance remain far below the ability of human experts.
  • 5.1.4 DeepScholar-ref: DeepScholar-ref improves over prior open-source systems and search agents, increasing six metrics by 1.06×–2.03× across five matched search-agent baselines.The gains are 1.18× Organization, 1.17× Nugget Coverage, 1.06× Relevance Rate, 2.03× Reference Coverage, 1.83× Citation Precision, and 1.86× Claim Coverage.
  • 5.2 Understanding Opportunities for Improvement.: Oracle retrieval nearly saturates Retrieval Quality and Verifiability for DeepScholar-ref, whereas standard retrievers score substantially lower, locating improvement opportunities in retrieval and synthesis.Human–LLM agreement was 71.43% for Organization, 83.33% for nugget labeling, and 65.9% for reference importance; reference coverage remains conservative because 24.2% of essential references were under-labeled.

6 Related Work

The paper distinguishes DeepScholar-bench from prior benchmarks by targeting live-web research synthesis rather than narrower long-form or short-form evaluation settings.

  • Prior expert-curated long-form research-synthesis benchmarks are expensive to construct and update, so they can become stale.
  • AcademicEval [63], LongBench-Cite [64], and SciIG [13] evaluate long-form generation without requiring live-web search.
  • Question-answering and factuality benchmarks emphasize short, easily verifiable answers, unlike complex research synthesis with multiple possible reasonable answers.

7 Conclusion

The paper concludes that DeepScholar-bench is a live, holistic benchmark for generative research synthesis, complemented by a reference pipeline and evidence that current systems remain far from saturated.

  • DeepScholar-bench automatically sources recent, high-quality ArXiv queries and evaluates retrieval quality, knowledge synthesis, and verifiability.
  • DeepScholar-ref provides a strong baseline for generative research synthesis.
  • No evaluated system surpasses a geometric mean of 31% across all metrics, demonstrating substantial room for improvement.
  • The benchmark’s difficulty and the observed performance gap motivate further advancement in generative research synthesis.

A Appendix

Figure 3 presents the DeepScholar-bench dataset schema.

  • Figure 3 presents the DeepScholar-bench dataset schema.
  • The appendix includes a figure devoted to the benchmark’s dataset schema.
  • The figure is identified as part of the DeepScholar-bench appendix material.

A.1 DeepScholar-bench Dataset

The appendix documents the DeepScholar-bench dataset through an overview, schema, and a table listing papers included in DeepScholar-June-2025.

  • A.1 DeepScholar-bench Dataset: The appendix provides a detailed overview and schema of the DeepScholar-bench dataset.
  • A.1 DeepScholar-bench Dataset: Table 5 lists the papers included in DeepScholar-June-2025.
  • A.1 DeepScholar-bench Dataset: The dataset documentation covers both structural schema information and the papers included in a dated benchmark slice.

A.2 Overview of Baselines and Experimental Setup

The evaluation covers open-source research systems, search agents using several language models, OpenAI’s o3-deep-research, and DeepScholar-ref. Reports are scored with LLM judges across synthesis and retrieval-quality metrics under specified aggregation and citation-window settings.

  • The benchmark compares open-source systems, model-augmented search agents, OpenAI’s o3-deep-research, and DeepScholar-ref.
  • GPT-4.1 judges Nugget Coverage, while GPT-4o judges Organization, Relevance Rate, Reference Coverage, Citation Precision, and Claim Coverage.
  • Nugget Coverage uses the strict all score, Organization uses win rate including ties, and Claim Coverage uses a window size of w = 1.

A.2.1 Open-source Research Systems

The open-source comparison includes DeepResearcher, STORM, and OpenScholar, evaluated primarily with the Llama-4-Scout-17B-16E-Instruct model. The systems represent agentic web research, multi-agent perspective discovery, and specialized literature retrieval, respectively.

  • DeepResearcher, STORM, and OpenScholar are the three evaluated open-source research systems.
  • DeepResearcher uses trained agents for web navigation, browsing, and synthesis, with its released agent trained by end-to-end reinforcement learning on Qwen2.5-7B-Instruct.
  • The evaluation reports DeepResearcher’s better-performing configuration, finding the Llama-4-Scout-17B-16E-Instruct backbone outperforms the released model.
  • STORM discovers diverse research perspectives through conversations among multiple agents and web documents before writing grounded, organized long-form articles.
  • OpenScholar combines a trained retriever over the 45-million-paper peS2o corpus with web search restricted to the ArXiv API.

A.2.2 Search Agents

The search-agent baselines pair several language models with ArXiv search through the ODS framework, enabling tool calls to the search API.

  • Search agents use Llama-4-Scout-17B-16E-Instruct, GPT-4.1, o3, Claude-opus-4, and Gemini-2.5-pro with ArXiv search access.
  • The ODS framework allows each language model to make tool calls to the ArXiv search API.

A.2.3 Commercial Systems.

The commercial comparison centers on OpenAI’s o3-deep-research, alongside DeepScholar-ref and robustness analyses across domains, query formulations, human judgments, and LLM judges. These analyses indicate generalization beyond Computer Science, strong query-type agreement, and substantial human–LLM metric agreement, while exposing sensitivity in selected metrics and citation-window trade-offs.

  • A.2.3 Commercial Systems.: OpenAI’s o3-deep-research is the commercial generative research synthesis system evaluated through its public API.
  • A.2.3 Commercial Systems.: DeepScholar-ref is evaluated with five model configurations, using the same or weaker models for semantic filtering and top-k operations across two search rounds.
  • A.3.2 Results on DeepScholar-Nov-2025: The November 2025 slice spans more than 75 ArXiv subject areas, and its results broadly match the June 2025 benchmark patterns.The o3-based variants outperform Llama-4-Scout variants on several synthesis and retrieval metrics, while Reference Coverage and Document Importance remain broadly low.
  • A.3.2 Results on DeepScholar-Nov-2025: The automated pipeline can generate new domain- or time-specific benchmark slices from query papers and date ranges under the same evaluation protocol.
  • A.3.2 Results on DeepScholar-Nov-2025: System scores show very strong agreement across query formulations, typically above 0.95 for most metrics and above 0.77 for Coverage Relevance Rate.Document Importance and Citation Precision are more sensitive to query phrasing, despite some large correlations not reaching statistical significance.
  • A.3.2 Results on DeepScholar-Nov-2025: LLM-based metrics achieve above 70% agreement with human annotations across the reported classification tasks.
  • A.4.2 Ablation Study on Verifiability: Increasing the citation window improves coverage, but larger windows reduce readability and make claim–citation verification harder, creating a precision–recall trade-off.Human exemplars average at least one citation per sentence, while w = 0 is stricter and w ≥1 is more recall-oriented.
  • A.4.3 Document Importance Across Human Exemplars: Citation-count distributions are highly skewed by papers exceeding 10k citations, with means of 478.3 overall and 647.6 for ArXiv-only references versus medians of 31 and 36.This skew limits citation counts as a proxy for importance.
Loading 2508.20033v2…