PaperlayerView exact revision on arXiv

Source-linked AI summary

ResearcherBench: Evaluating Deep AI Research Systems on the Frontiers of Scientific Inquiry

Tianze Xu, Pengrui Lu, Lyumanshan Ye, Xiangkun Hu, Pengfei Liu

arXiv:2507.16280v1cs.AI

TL;DR

Existing benchmarks mainly assess retrieval and synthesis of established knowledge, not DARS’ ability to assist with open-ended frontier questions. ResearcherBench combines curated frontier-AI questions with rubric-based and factual assessment. The authors report high agreement between human annotators and the selected rubric judge model.

  • Problem

    Existing benchmarks focus on retrieval and synthesis of established knowledge, leaving DARS’ ability to assist with open-ended frontier questions insufficiently evaluated.

    Source: §1 · p. 2

  • Method

    ResearcherBench curates authentic frontier-AI questions and combines expert-designed rubric assessment with factual assessment of faithfulness and groundedness.

    Source: §1 · p. 2, §3 · p. 4, §4 · p. 5

  • Results

    The weighted F1-score of 0.80 for o3-mini falls within the reported “high agreement” range for AI evaluation versus expert annotation comparisons.

    Source: §6 · p. 10

  • Takeaways & Limitations

    The authors frame involving DARS in challenging frontier AI research as a potential feedback loop for recursive self-improvement.

    Source: §1 · p. 2

    The 65-question dataset is a relatively small sample that may not capture the full spectrum of frontier research challenges.

    Source: §A.1 · p. 12

Abstract

from arXiv · show

The emergence of deep research systems presents significant capabilities in problem-solving, extending from basic queries to sophisticated research tasks. However, existing benchmarks primarily evaluate these systems as agents for web retrieval and report generation, overlooking their potential to discover novel insights on the frontiers of scientific research. To address this gap, we introduce ResearcherBench, the first benchmark focused on evaluating the capabilities of these advanced, agentic systems - which we refer to as Deep AI Research Systems (DARS) - on frontier AI scientific questions. We compiled a dataset of 65 research questions expertly selected from real-world scientific scenarios such as laboratory discussions and interviews, spanning 35 different AI subjects and categorized into three types: technical details, literature review, and open consulting. Our dual evaluation framework combines rubric assessment, which uses expert-designed criteria to evaluate insight quality, with factual assessment, which measures citation accuracy (faithfulness) and coverage (groundedness). We evaluated several leading commercial DARS and baseline systems. Results show that OpenAI Deep Research and Gemini Deep Research significantly outperform other systems, with particular strength in open-ended consulting questions. Such capabilities represent a meaningful step toward AI self-improvement, aligning with the vision of ASI for AI. We open-source ResearcherBench to provide a standardized platform for promoting the development of next-generation AI research assistants, hoping to foster a new perspective in AI research evaluation for a novel pattern of scientific collaboration: https://github.com/GAIR-NLP/ResearcherBench.

1. Introduction

Existing benchmarks emphasize retrieving and synthesizing established knowledge, leaving DARS’ ability to assist with unresolved frontier questions insufficiently evaluated. ResearcherBench addresses this gap with frontier-AI questions and complementary assessments of insight quality and factual reliability.

1. Introduction

  • The paper presents frontier research assistance as a possible route toward recursive AI self-improvement and the broader vision of ASI for AI.It describes a potential feedback loop in which AI helps accelerate its own development.Source: §1 · p. 2
  • ResearcherBench evaluates whether DARS can provide meaningful assistance on unsolved, cutting-edge AI research problems rather than only retrieve and synthesize established information.The paper frames this as a test of whether systems can act as research partners by understanding complex problems and generating insights.Source: §1 · p. 2
  • The benchmark contains 65 high-quality questions from authentic frontier research scenarios, spanning 35 AI subjects and three research-assistance types.Sources include laboratory discussions, interviews with AI researchers, and active scientific forums.Source: §1 · p. 2
  • Its dual evaluation framework pairs expert-crafted rubric criteria for insight quality with factual assessment of faithfulness and groundedness.Faithfulness measures factual accuracy, while groundedness measures coverage of generated content.Source: §1 · p. 2

2. Related Work

Related work describes the shift toward agentic systems with iterative research workflows and benchmarks for retrieval, synthesis, and web interaction. The paper argues that these evaluations do not fully capture frontier-research insight, and reports findings from its evaluation alongside an open-source release.

2. Related Work

  • The analysis identifies exploratory reasoning on open-ended tasks as a primary DARS strength, rather than precise technical or literature synthesis.It also reports a “high faithfulness, low groundedness” pattern and finds that citation coverage does not necessarily correspond to superior insight quality.Source: §2 · p. 3
  • The authors release ResearcherBench’s curated questions and dual evaluation framework as standardized infrastructure for benchmarking DARS.They intend the platform to support collaborative development of AI systems for scientific research assistance.Source: §2 · p. 3

2.1 Deep AI Research Systems

  • Modern DARS use autonomous query formulation, iterative reflection, and dynamic search refinement, while earlier approaches relied on static repositories and lacked iterative reasoning.Commercial implementations are described as supporting multi-step browsing, research planning, and report generation.Source: §2.1 · p. 3

2.2 Research-related Evaluation Benchmarks

  • Existing benchmarks assess report generation, literature synthesis, web interaction, or replication, but the paper argues they miss expert-judged insight on ambiguous frontier problems.The authors say frontier rubrics require domain expertise and nuanced judgments that cannot be reliably generated directly by language models.Source: §2.2 · p. 3

3. ResearcherBench

ResearcherBench curates authentic frontier-AI research questions from real-world scientific settings and organizes them for systematic evaluation. Its composition spans diverse AI subjects and three assistance types, while the selection process filters an initial collection of several hundred candidate problems.

3 ResearcherBench

  • The benchmark construction combines systematic collection and filtering to produce 65 authentic questions across 35 AI subjects.Figure 2 presents the subject distribution and representative question examples.Source: §3 · p. 4, Figure 2 · p. 4

3.1 Data Collection Strategy

  • Questions were collected from laboratory discussions, interviews with leading AI researchers, and scientific forums where practitioners discuss challenges and gaps.The initial corpus contained several hundred organically arising candidate questions rather than artificially constructed queries.Source: §3.1 · p. 4

3.2 Dataset Composition and Categorization

  • Claude-3.7-Sonnet classified questions into technical details, literature review, and open consulting to evaluate distinct research-assistance scenarios.The paper presents this categorization as a way to assess DARS across different cognitive demands.Source: §3.2 · p. 4
  • The 35 subjects include model architecture, multimodal fusion, AI ethics, and emerging machine-learning paradigms.The authors characterize this topical distribution as covering scenarios across frontier AI research.Source: §3.2 · p. 4

3.3 Question Selection and Filtering

  • To construct the benchmark, the authors designed a multidimensional question-selection framework to filter high-quality problems from several hundred initial candidates.The supplied passage does not specify the framework’s dimensions.Source: §3.3 · p. 4

4. Dual Evaluation Framework

ResearcherBench pairs expert-designed rubric assessment of response quality with factual assessment of citation accuracy and groundedness. Its rubrics are built and checked with domain-expert input, while factual evaluation extracts claims and their source links for verification.

4. Dual Evaluation Framework

  • 65 questions across 35 AI research subjects are grouped into technical details, literature review, and open consulting, emphasizing verification, synthesis, and broader insights, respectively.Experienced AI researchers and practitioners reviewed candidate questions against selection criteria.Source: §4 · p. 5, Table 1 · p. 5
  • The framework combines expert-designed rubric assessment of insight quality with factual assessment of citation accuracy and groundedness.The two components target response quality and reliability, respectively.Source: §4 · p. 5, §4.2 · p. 6, §4.2.1 · p. 6

4.1 Rubric Assessment: Expert-designed Criteria Evaluation

  • Frontier-question assessment targets conceptual depth, theoretical understanding, and insight quality that simple LLM-as-a-Judge scoring may not capture.The rubric decomposes complex questions into weighted, evaluable components addressing conceptual understanding, methodological rigor, and analytical depth.Source: §4.1 · p. 5, §4 · p. 5

4.1.1 Criteria Construction

  • Rubric construction combines multi-source insight extraction, expert-designed weighted rubric items, and iterative quality checks.Experts validate synthesized reference materials, collaboratively review rubrics, pilot-test items, and compare automated assessment with human judgments.Source: §4.1.1 · p. 6

4.1.2 Evaluation Metric

  • A weighted coverage score aggregates binary judgments of whether each response addresses each expert-designed rubric item.Each item's weight reflects its relative importance; the score accounts for both criterion coverage and importance.Source: §4.1.2 · p. 6

4.2 Factual Assessment: Faithfulness and Groundedness Evaluation

  • Factual assessment evaluates citation accuracy and content groundedness as complementary reliability dimensions for generated research reports.The framework is described as a three-step automated process.Source: §4.2 · p. 6, §4.2.1 · p. 6

4.2.1 Assessment Methodology

  • Claim extraction captures factual claims with context and checks whether each has a retrievable citation URL for subsequent verification.Claims without a corresponding citation URL are treated as ungrounded and assigned an empty URL.Source: §4.2.1 · p. 6

5. Experiments and Results

ResearcherBench evaluates both the insight quality and factual reliability of deep research systems on frontier questions. OpenAI Deep Research and Gemini Deep Research lead rubric assessment, while results also reveal gaps between citation accuracy, coverage, and research quality.

5. Experiments and Results

  • The factual assessment extracts claims and citation URLs, checks source support with a binary judge, and measures citation accuracy and claim coverage.Context passages can help the judge assess semantically incomplete or ambiguous claims; faithfulness concerns cited claims, while groundedness concerns all factual claims.Source: §5 · p. 7

5.1 Experimental Setup

  • The evaluation compares four commercial deep research systems and two web-search LLM baselines; data collection ran from March to April 2025.o3-mini judges rubric coverage, while GPT-4.1 serves as both extraction and judge model for factual assessment.Source: §5.1 · p. 7, §5.2 · p. 8

5.2.1 Overview

  • 20–30% advantages on rubric assessment distinguish OpenAI Deep Research and Gemini Deep Research from other DARS.Perplexity Deep Research performs moderately, while Grok3 DeepSearch & DeeperSearch perform relatively poorly.Source: Table 2 · p. 8, §5.2.1 · p. 8
  • Most DARS score above 0.8 in faithfulness, yet groundedness is low; Gemini Deep Research achieves the best balance between the two metrics, reflecting superior citation strategy optimization.Source: §5.2.1 · p. 8

5.3 Key Findings

  • Dedicated DARS outperform LLMs with basic web search, while Perplexity Sonar Reasoning Pro approaches Perplexity Deep Research and exceeds Grok3.The authors note that web search alone is insufficient, while combining deep reasoning with search can be competitive to some extent.Source: §5.3 · p. 8
  • High citation coverage does not necessarily correspond to higher research quality: OpenAI leads rubric assessment at 0.34 groundedness, while Sonar Reasoning Pro leads groundedness at 0.68.The authors suggest that valuable frontier-research insights may emerge through synthesis and reasoning beyond direct source attribution.Source: §5.3 · p. 8, §5.4 · p. 9

5.4 Judge Model Selection and Meta Evaluation

  • All systems perform better on open consulting than on technical details or literature review, with top systems exceeding 76% coverage.Gemini performs best on technical details; OpenAI leads on open consulting and literature review.Source: Figure 3 · p. 9, §5.4 · p. 9
  • Judge-model meta-evaluation compares automated rubric assessments with expert judgments on 10 responses, using agreement metrics and API cost as selection criteria.Researchers assessed the responses with the same rubrics used in automated evaluation.Source: §5.4 · p. 9, Table 3 · p. 9

6. Conclusion

The paper presents ResearcherBench as a benchmark for frontier AI research questions, with a dual evaluation framework. It reports strong agreement between human annotators and selected judge models and argues that open-sourcing the benchmark can prioritize insight generation over information coverage.

6. Conclusion

  • o3-mini was selected for rubric assessment with F1-score 0.80, precision 0.85, and evaluation cost $0.13 per question; GPT-4.1 was selected for factual assessment.GPT-4.1 was chosen for its long-context processing performance in claim extraction and citation-support evaluation.Source: §6 · p. 10
  • 0.80 weighted F1-score for o3-mini falls within the stated “high agreement” range for AI evaluation against expert annotation.The authors use this agreement to support the reliability of their rubric-based assessment framework.Source: §6 · p. 10

6 Conclusion

  • ResearcherBench contributes a curated dataset of 65 frontier questions across 35 AI subjects and three question types, alongside rubric and factual assessment.The question types are technical details, literature review, and open consulting.Source: §6 · p. 10
  • Open-sourcing ResearcherBench is intended to prioritize depth of understanding and insight generation over breadth of information coverage.The authors frame this as a shift toward evaluating DARS as research partners.Source: §6 · p. 10

A.1 Limitations

This section discusses limits on generalizing ResearcherBench’s findings and understanding system performance, alongside plans for broadening and updating the benchmark and selecting questions and criteria. It also describes a pipeline for deriving weighted, verifiable rubrics from documents and distinguishing document-based information from any permitted general knowledge.

A.1 Limitations

  • The benchmark covers only AI research and 65 questions, limiting cross-domain generalizability and coverage of frontier challenges; commercial black-box systems also obscure the sources of performance differences.The 65-question set comprises 12 technical-details, 20 literature-review, and 33 open-consulting questions.Source: §A.1 · p. 12, §A.2 · p. 13

A.2 Future Works

  • Cross-domain expansion to fields such as biology, chemistry, physics, and social sciences could test whether observed open-consulting superiority generalizes beyond AI research.Source: §A.2 · p. 13
  • Periodic question updates with active-researcher input and longitudinal evaluation of new DARS releases could keep the benchmark current and track capability changes over time.Source: §A.2 · p. 13

B.1 Question Selection Framework

  • Questions were classified into technical-details, literature-review, and open-consulting types, scored with type-specific rubrics by at least two researchers, and retained at average scores of 4.0 or higher.Disagreements were resolved through discussion, yielding 65 questions from an initial collection of several hundred problems.Source: §B.1 · p. 13

B.2 Question Evaluation Rubrics

C. Guidelines for Rubric Design SII-GAIR

  • For open consulting, the rubric values questions that encourage multi-perspective exploration, address future challenges, inspire novel ideas, and remain grounded and relevant beyond transient trends.Source: C. Guidelines for Rubric Design SII-GAIR · p. 15

C.1 Key Insight Extraction

  • Claude-3.7-Sonnet analyzed question context and generated reference materials; open-consulting insight guidance emphasized novel insights, critical analysis, and new hypotheses or conceptual frameworks.Source: §C.1 · pp. 15–16

C.1 Key Insight Extraction SII-GAIR

  • The prompt requests 8–15 key insights representing the question’s most valuable aspects, with each insight substantive, directly relevant, expert-level or specialized, and specific enough to be evaluable.Source: §C.1 · p. 17
  • The answer prompt prioritizes the document and extracted insights, allowing supplemental general knowledge only when needed and clearly identified as such.It also asks analysts to organize relevant information coherently and accurately.Source: §C.1 · p. 17

C.1.2 Human Verification and Rubric Design

  • Human annotators convert key insights and reference materials into question-specific rubrics, assigning each item a weight from 1–3 according to its importance.Core items receive weight 3, supporting items 2, and nice-to-have items 1; total weights should reflect the hierarchy.Source: §C.1.2 · p. 18
  • Rubric items should be clear, independent, concept-focused, evaluative, action-oriented, and specific enough for objective pass/fail assessment.Quality checks require binary assessment, avoiding subjective wording, testing items against reference materials, and collectively covering essential answer aspects.Source: §C.1.2 · p. 18

D Experimental Setup

The evaluation used system-specific default interaction settings and adjusted web-search context for non-DARS systems. Most DARS interactions were conducted through web user interfaces.

D Experimental Setup

  • For most DARS systems, interaction through web user interfaces limited evaluation scalability; Perplexity Deep Research was the exception.The authors used default settings, including standard replies to OpenAI Deep Research follow-up questions and unmodified Gemini-2.5-Pro Deep Research plans.Source: D Experimental Setup · p. 18
  • GPT-4o Search Preview and Perplexity: Sonar Reasoning Pro used “High” Search Context Size to support comparison with dedicated research agents.The passage says this was intended to compensate for the lack of specialized research workflows.Source: D Experimental Setup · p. 18

E. Data Collection Timeframes SII-GAIR

The report-generation prompt was designed to guide responses from non-DARS systems. It requested a structured, detailed report with academic references and numbered IEEE-style citations.

E. Data Collection Timeframes SII-GAIR

E Data Collection Timeframes

The appendix documents DARS data-collection timeframes and specifies prompts for rubric coverage, factual-claim extraction, and reference verification.

E Data Collection Timeframes

  • Collection timeframes are documented because commercial DARS model iterations and technical details are often opaque, supporting reproducibility and analysis of temporal effects on performance.Table 4 is titled “Data Collection Timeframes for DARS,” but the supplied passage does not give its entries.Source: E Data Collection Timeframes · p. 19, Table 4 · p. 19

Rubric Coverage Evaluation Prompt

Claims Extraction Prompt

  • The claims-extraction prompt requests distinct factual claims and returns each claim with its context and a citation URL when a citation marker directly follows it.Source: Claims Extraction Prompt · p. 20

F.4 Claim Verification Prompt SII-GAIR

  • Extraction guidelines retain claims whether cited or not, map citation markers to URLs only when they directly follow a claim, and exclude opinions and methodological descriptions.Source: §F.4 · p. 21

Claim Verification Prompt

  • The verification prompt asks whether reference content supports each claim and requires a yes, no, or unknown result in the specified JSON format.Source: Claim Verification Prompt · p. 21

F.4 Claim Verification Prompt SII-GAIR

  • Verification uses the supplied reference content rather than general knowledge; context can resolve ambiguity, while unresolved or unavailable references yield unknown.Source: §F.4 · p. 22

F.1 Implementation Details

Implementation choices simplify rubric scoring and factual verification while addressing efficiency, context limits, and retrieval errors.

Rubric Assessment Implementation

  • Rubric items are evaluated independently using the prompt template provided in Appendix F.2.Source: §F.1 · p. 19

Factual Assessment Implementation

  • For factual assessment, reports are segmented at paragraph boundaries and claims are extracted section by section before being aggregated into a complete response-level claim set.This section-based strategy addresses computational efficiency and context-length limits.Source: §F.1 · p. 19
  • Claims sharing a URL are grouped for batch verification, and an unknown judgment handles extraction or retrieval problems; unknown claims are excluded from final metrics.Source: §F.1 · p. 19
Loading 2507.16280v1…