PaperlayerView exact revision on arXiv

Source-linked AI summary

DeepResearch Arena: The First Exam of LLMs' Research Abilities via Seminar-Grounded Tasks

Haiyuan Wan, Chen Yang, Junchi Yu, Meiqi Tu, Jiaxuan Lu, Di Yu, Jianbao Cao, Ben Gao, Jiaqing Xie, Aoran Wang, Wenlong Zhang, Philip Torr, Dongzhan Zhou

arXiv:2509.01396v2cs.AI

TL;DR

Faithfully evaluating deep-research agents is difficult because high-quality frontier research questions are hard to collect. DeepResearch Arena builds traceable tasks from academic seminars and evaluates factual grounding and higher-order reasoning; its evaluations show substantial differences across models and task types.

  • Problem

    Faithfully evaluating deep-research agents remains difficult because collecting high-quality frontier research questions that capture researchers’ interests is challenging.

    Source: Introduction · p. 1

  • Method

    DeepResearch Arena draws on seminars and uses MAHTG to turn research inspirations into traceable tasks, with factual-grounding and rubric-based reasoning evaluation.

    Source: Introduction · pp. 1–2

  • Results

    Models differ substantially across research task types; gpt-o4-mini-deepresearch and gemini-2.5-flash are consistently strong, and gpt-o4-mini-deep-research achieves the highest ACE score of 4.03.

    Source: Experiments · pp. 6–7

  • Takeaways & Limitations

    DeepResearch Arena offers a theory-aligned foundation for assessing deep-research agents across factual grounding and higher-order reasoning.

    Source: Conclusion · p. 7

Abstract

from arXiv · show

Deep research agents have attracted growing attention for their potential to orchestrate multi-stage research workflows, spanning literature synthesis, methodological design, and empirical verification. Despite these strides, evaluating their research capability faithfully is rather challenging due to the difficulty of collecting frontier research questions that genuinely capture researchers' attention and intellectual curiosity. To address this gap, we introduce DeepResearch Arena, a benchmark grounded in academic seminars that capture rich expert discourse and interaction, better reflecting real-world research environments and reducing the risk of data leakage. To automatically construct DeepResearch Arena, we propose a Multi-Agent Hierarchical Task Generation (MAHTG) system that extracts research-worthy inspirations from seminar transcripts. The MAHTG system further translates research-worthy inspirations into high-quality research tasks, ensuring the traceability of research task formulation while filtering noise. With the MAHTG system, we curate DeepResearch Arena with over 10,000 high-quality research tasks from over 200 academic seminars, spanning 12 disciplines, such as literature, history, and science. Our extensive evaluation shows that DeepResearch Arena presents substantial challenges for current state-of-the-art agents, with clear performance gaps observed across different models.

Introduction

DeepResearch Arena addresses the difficulty of faithfully evaluating deep-research agents by drawing open-ended tasks from academic seminars. Its MAHTG pipeline turns seminar inspirations into traceable tasks, alongside a hybrid evaluation of factual grounding and higher-order reasoning.

  • Existing benchmarks use static corpora or manually curated tasks, risking leakage, limited scale and diversity, and distance from evolving real-world inquiry.Source: Introduction · p. 1
  • DeepResearch Arena uses seminar discourse, where researchers develop uncertain questions through live discussion, as a more faithful research setting with lower pretraining-leakage risk.Source: Introduction · p. 1
  • MAHTG filters and structures research-worthy inspirations into traceable open-ended tasks while preserving the intellectual context of expert discourse.Source: Introduction · p. 2
  • The benchmark spans over 10,000 tasks and pairs factual-alignment metrics with rubric-based evaluation of open-ended reasoning.Source: Introduction · p. 2

Related Works

Prior deep-research agents use tool-supported, multi-stage workflows, while existing benchmarks draw mainly on static corpora or expert-authored tasks. Both benchmark approaches have limitations in capturing how research questions emerge through authentic discourse.

  • Tool-enabled deep research agents support multi-stage workflows and exploratory, open-ended inquiry, extending beyond retrieval-based assistance.Source: Related Works · p. 2
  • Static-corpus benchmarks often test predefined reasoning paths and factual retrieval, but do not capture how research questions evolve and iterate in real contexts.Source: Related Works · pp. 2–3
  • Expert-authored benchmarks offer original, challenging, discipline-specific tasks, but manual construction limits scalability and diversity and misses questions emerging through discourse and ambiguity.Source: Related Works · p. 3

Multi-Agent Hierarchical Task Generation.

MAHTG transforms seminar recordings into research tasks through transcript processing, inspiration extraction, task synthesis, and Elo-based ranking.

Data Collection

Inspiration Extraction

Task Generation and Filtering

  • TaskWeaver aggregates inspirations into focused tasks across Synthesize, Design, and Evaluate, pairing core problems or methodological cues with executable goals.Source: Multi-Agent Hierarchical Task Generation · p. 4
  • RankEval initializes tasks at 1200 Elo and compares disjoint pairs on originality, clarity, and relevance, using confidence-weighted outcomes before selecting top-rated tasks.Scores update over repeated rounds of comparisons.Source: Multi-Agent Hierarchical Task Generation · p. 4

Evaluation Methodology

The evaluation framework pairs factual grounding against reference keypoints with rubric-based scoring of open-ended research outputs. Together, KAE and ACE assess distinct dimensions of research-agent performance.

Evaluation Methodology

  • The hybrid framework combines Keypoint-Aligned Evaluation (KAE) for factual grounding with Adaptively-generated Checklist Evaluation (ACE) for open-ended reasoning.It is designed to assess performance across research stages, from literature synthesis to hypothesis generation and empirical validation.Source: Evaluation Methodology · p. 4
  • KAE retrieves cited webpages, extracts their factual keypoints, and consolidates them into a unified, de-duplicated reference set.The report is evaluated against keypoints drawn from the URLs it cites.Source: Evaluation Methodology · pp. 4–5
  • KAE measures the proportion of reference keypoints supported, contradicted, or omitted in a generated report.High support and low conflict and omission indicate stronger factual alignment.Source: Evaluation Methodology · p. 5
  • ACE first generates a task-specific weighted checklist, then uses a separate model to score each criterion and aggregate a task-level rating.Checklist criteria can cover factual correctness, methodological soundness, formatting, and reasoning clarity.Source: Evaluation Methodology · p. 5
  • ACE addresses open-ended tasks that lack fixed answers, while offering an alternative to hand-crafted criteria that are difficult to scale and generalize.The authors characterize ACE as flexible, scalable, and more reliable for nuanced research-task evaluation.Source: Evaluation Methodology · pp. 5–6

Experiments

The experiments compare research agents across evaluation dimensions and task types, finding distinct strengths in overall quality, factual alignment, and efficiency. Results also vary by task and language.

Implementation Details

  • MAHTG uses specialized agents and a heterogeneous model configuration, and the evaluation selects the top 100 highest-scoring samples from the full dataset.The stated selection is made to reduce evaluation costs.Source: Experiments · p. 6

Experimental Setup

  • The evaluation suite includes frontier research agents and retrieval-augmented models, and uses gemini-2.5-flash as a unified evaluator for factual and subjective scoring.The evaluated model set includes gpt-4o and gpt-4.1 search variants, o4-mini-deepresearch, Gemini models, and grok-4.Source: Experiments · p. 6

Overall Performance

  • gpt-o4-mini-deep-research has the highest ACE score at 4.03, while GPT-4.1 excels in factual precision and grok-4 has English KSR 83.3.Gemini-2.5-flash also scores strongly on factual coverage and low conflict and omission, but uses significantly more tokens; grok-4's coverage drops sharply in Chinese.Source: Experiments · p. 6

Performance on Different Tasks

  • gpt-o4-mini-deepresearch and gemini-2.5-flash perform strongly across nearly all task types, while the gpt-4o family struggles with multi-step logic and structured outputs.Figure 3 compares performance across 12 disciplines and 10 research task types.Source: Experiments · pp. 6–7, Figure 3 · p. 7

KAE Performance

  • Gemini-2.5-flash and gpt-o4-mini-deepresearch achieve the strongest overall KAE results, with high keypoint coverage, low conflict and omission, and the highest efficiency scores.By contrast, gemini-2.5-pro, gpt-4o-search-preview, and gpt-4.1-mini show higher conflict and omission and the lowest efficiency.Source: Experiments · p. 7, Figure 4 · p. 7

Data Leakage

  • The benchmark's effectiveness for preventing data leakage is evaluated in the appendix.Source: Experiments · p. 7

Conclusion

DeepResearch Arena combines a multidisciplinary seminar corpus, hierarchical task generation, and hybrid evaluation to assess research capabilities in realistic, open-ended settings. A leakage simulation tests whether evaluated models can reproduce withheld task text from a supplied prefix.

Conclusion

  • The benchmark assesses LLM-based agents across three essential stages using multidisciplinary seminars, hierarchical task generation, and hybrid evaluation of factual grounding and higher-order reasoning.The conclusion presents it as a theory-aligned foundation for advancing research assistants.Source: Conclusion · p. 7

A Data Leakage Detection

  • A leakage simulation prompts each of eight evaluated models with the first half of a task and checks whether it can reproduce the withheld portion.The task is split at the punctuation boundary closest to its midpoint.Source: A Data Leakage Detection · p. 10, §A.1 · p. 10

A.1 Experimental Procedure

  • Unexpectedly high similarity between a generated continuation and the withheld suffix may indicate that a model memorized or encountered the full task during pretraining.The authors frame this as a potential data-leakage risk.Source: §A.1 · p. 10

A.2 Similarity Metrics

The leakage check compares generated and reference task suffixes using three lexical similarity measures and a weighted composite. A composite score above the chosen threshold flags a task as potentially leaked.

A.2 Similarity Metrics

  • Normalized LCS string similarity compares generated and reference suffixes by their longest common character subsequence, symmetrically normalized by their average lengths.The score ranges from 0 to 1, with 1 indicating identical sequences and 0 indicating no character-level overlap.Source: §A.2 · p. 10
  • TF-IDF cosine similarity measures the angular similarity between fixed-vocabulary weighted bag-of-words representations of generated and reference suffixes, from 0 to 1.A score of 1 indicates identical vector direction.Source: §A.2 · p. 10
  • Word overlap measures the proportion of unique reference words recovered in the generated suffix, regardless of word order or repetition.The word sets are tokenized and lowercased before comparison.Source: §A.2 · p. 10
  • The composite similarity score combines three individual metrics in a weighted combination (De Boom et al. 2015): character-level string alignment, TF-IDF cosine similarity, and word-level lexical overlap.Source: §A.2 · p. 11

A.3 Leakage Criterion

  • A task is flagged as potentially leaked when its composite similarity exceeds the threshold τ = 0.7.The threshold is described as a conservative criterion intended to identify potential memorization or contamination while minimizing false positives.Source: §A.3 · p. 11

A.4 Experimental Setup

The leakage experiment evaluates eight models on the same 100 benchmark tasks under fixed decoding and output-length settings. A worked example illustrates a continuation that remains distinct from its reference suffix.

A.4 Experimental Setup

  • Eight models were evaluated on the same 100 tasks via external APIs, using temperature 0.1 and a maximum output length of 500 tokens.For each model-task pair, the authors logged the continuation, reference suffix, similarity scores, and leakage flag.Source: §A.4 · p. 11
  • In the non-leaked example, the model continuation differs from the reference suffix, with string similarity 15.3%, TF-IDF similarity 9.4%, word overlap 14.3%, and composite similarity 12.7%.The example is presented as a case below the leakage threshold.Source: §A.4 · p. 11

A.5 Results and Analysis

None of the 100 sampled tasks triggered the leakage criterion across the evaluated models. A separate section motivates checking whether automated scoring metrics align with expert judgments.

A.5 Results and Analysis

  • None of the 100 sampled tasks triggered the leakage criterion across the evaluated models; no model exceeded the composite similarity threshold of 0.7.Table 4 presents the detection summary for all eight models.Source: §A.5 · p. 11

B.1 Motivation

  • The authors motivate a systematic analysis of whether the automated scoring metrics KAE and ACE agree with expert annotations.The supplied passage states the analysis objective, not its outcome.Source: §B.1 · p. 11

B.2 Experimental Setup

The setup compares automated evaluation with averaged human judgments and assesses possible task leakage through suffix similarity.

B.2 Experimental Setup

  • Three annotators independently rate each response, and their scores are averaged for comparison with automated scores.Human evaluations cover a randomly sampled representative subset of benchmark tasks.Source: §B.2 · p. 11
  • Table 4 compares suffix similarity across models over 100 tasks each and flags composite similarity above 0.7 as potential leakage.The caption defines Count as suspected leakage cases and reports all counts as zero.Source: Table 4 · p. 12
  • Spearman, Pearson, and Kendall correlations measure agreement between automated scores and averaged human scores.They capture ranked monotonic relationship, linear correlation, and ordinal association, respectively.Source: §B.2 · p. 12

B.3 Results and Analysis

KAE and ACE show strong agreement with human judgments, especially in ranking consistency; a qualitative example illustrates ACE alignment with human intuition, and the authors conclude both metrics support scalable evaluation.

B.3 Results and Analysis

  • Both KAE and ACE correlate strongly with human scores, particularly in ranking consistency.The section reports this agreement in Table 5 for a representative set of benchmark tasks.Source: Table 5 · p. 12, §B.3 · p. 12

B.4 Qualitative Case Study

  • A representative case illustrates how ACE aligns with human intuition on a realistic task.The example is presented in Table 6.Source: §B.4 · p. 12

B.5 Summary

  • The authors conclude that KAE and ACE offer scalable, interpretable, human-aligned evaluation without manual annotation’s cost and variability.Source: §B.5 · p. 12

C Sample Checklist Generated by ACE

This appendix documents ACE-generated, task-specific checklists and the prompt templates used to construct research tasks and evaluate responses, including weighted query-specific criteria, calibrated rating bands, and structured JSON output.

Generated Checklist

  • For a hypergraph-clustering design task, ACE’s example checklist weights methodological rigor at 0.45, technical accuracy at 0.30, completeness at 0.15, and clarity at 0.10.The task asks for an experimental protocol comparing greedy maximal independent set algorithms with edge-colored clustering methods, including datasets, complexity validation, and significance testing.Source: Generated Checklist · p. 13

D Prompt Templates

  • The prompt templates cover inspiration extraction, task generation and scoring, evidence extraction and relevance assessment, and checklist construction and criterion scoring.They standardize interactions across models and tasks, and are described as supporting benchmark transparency and reproducibility.Source: D Prompt Templates · p. 14

INSPIRATION EXTRACTION PROMPT

  • The inspiration-extraction prompt screens transcript leads for at least two of four qualities, assigns one of four categories, and directs scanning, clustering, qualification, and faithful paraphrase.The qualities are novelty, explorability, challenge, and verifiability; the categories are Limitation, Methodology, Transdisciplinary, and Hypothesis.Source: INSPIRATION EXTRACTION PROMPT · p. 15

TASK GENERATOR PROMPT

  • The task generator returns 5–8 self-contained tasks across Synthesize, Design, and Evaluate, grounding them in inspirations and requiring concrete deliverables.It also steers task emphasis by inspiration type and distinguishes Basic tasks feasible with public data in three hours or less from Advanced tasks needing novel data, tools, or reasoning.Source: TASK GENERATOR PROMPT · p. 16

RESEARCH TASK SCORING PROMPT

  • The task-scoring prompt compares two tasks on clarity, actionability, novelty, depth-fit, and consistency, scores each dimension from 1 to 5, and selects the higher overall score.For a tie, it chooses the slightly better task and sets confidence to 0.55.Source: RESEARCH TASK SCORING PROMPT · p. 17

KEY POINT EXTRACTION PROMPT

  • The key-point extraction prompt identifies query-relevant evidence and requires exact source-text spans, while allowing concise or abstract point descriptions.It specifies that every span must be copied verbatim from the supporting passage.Source: KEY POINT EXTRACTION PROMPT · p. 18

KEY POINT RELEVANCE PROMPT

  • The relevance prompt classifies whether a response SUPPORTS, OMITS, or CONTRADICTS a key point and requires confidence, reasoning, and key aspects.The judgment considers the original task, response content, and key point.Source: KEY POINT RELEVANCE PROMPT · p. 19

CHECKLIST TEMPLATE PROMPT (Page 1/2)

  • The checklist template requires actionable, query-specific, weighted criteria, with 3–6 items whose weights vary by phase and task type.Its phase-specific priorities include methodological rigor and feasibility for experiment blueprints, statistical rigor for empirical tests, and fairness for comparative analyses.Source: CHECKLIST TEMPLATE PROMPT (Page 1/2) · p. 20

CHECKLIST TEMPLATE PROMPT (Page 2/2)

  • The checklist template assigns four query-specific criteria weights of 0.4, 0.3, 0.2, and 0.1, requiring the top criterion to match the task’s critical requirement.It also requires non-generic descriptions, weights summing to approximately 1.0, and directly parseable valid JSON.Source: CHECKLIST TEMPLATE PROMPT (Page 2/2) · p. 21

SINGLE CRITERION SCORING PROMPT (Page 1/2)

  • The prompt evaluates a single checklist criterion against top-tier-venue exacting standards, listing task category, type, and difficulty, and rating exceptional mastery 8–10, basic competence 5–7, and inadequacy 1–4.Source: SINGLE CRITERION SCORING PROMPT (Page… · p. 22

SINGLE CRITERION SCORING PROMPT (Page 2/2)

  • The scoring prompt tests criterion coverage, treatment quality, depth, rigor, professional adequacy, and gaps, while directing evaluators to assume inadequacy and actively seek weaknesses.It requests a 0–10 integer rating and justification in valid JSON, reserving high scores for exceptional criterion-specific performance.Source: SINGLE CRITERION SCORING PROMPT (Page… · p. 23

Related papers

Loading 2509.01396v2…