PaperlayerView exact revision on arXiv

Source-linked AI summary

PaperScope: A Multi-Modal Multi-Document Benchmark for Agentic Deep Research Across Massive Scientific Papers

Lei Xiong, Huaying Yuan, Zheng Liu, Zhao Cao, Zhicheng Dou

arXiv:2604.11307v1cs.AI

TL;DR

Multi-modal, multi-document scientific reasoning remains underrepresented, despite its importance to research workflows that integrate evidence across papers. PaperScope builds a graph-guided benchmark with diverse research tasks, and evaluations find substantial limitations among current agents, including in retrieval and multi-source integration.

  • Problem

    Existing benchmarks focus mainly on single-document understanding, leaving multi-modal, multi-document scientific reasoning underrepresented despite its role in research workflows.

    Source: §1 · p. 1

  • Method

    PaperScope builds a graph-grounded multi-modal benchmark by extracting and linking scientific-paper information, selecting related papers with ORWAS, and constructing diverse research questions.

    Source: §1 · pp. 1–2, §4.1 · p. 5

  • Results

    Current agents show substantial limitations in large-scale document research, including insufficient accuracy and weaknesses in multimodal understanding, retrieval, and multi-source information integration.

    Source: §1 · p. 2

  • Takeaways & Limitations

    PaperScope provides a benchmark and scalable construction pipeline for evaluating multi-source, multi-modal scientific deep research.

    Source: §1 · p. 2

    The benchmark uses a finite local corpus, heuristic article selection without global-optimality guarantees, LLM-based generation, and limited baseline coverage under fixed inference budgets.

    Source: Limitations · p. 10

Abstract

from arXiv · show

Leveraging Multi-modal Large Language Models (MLLMs) to accelerate frontier scientific research is promising, yet how to rigorously evaluate such systems remains unclear. Existing benchmarks mainly focus on single-document understanding, whereas real scientific workflows require integrating evidence from multiple papers, including their text, tables, and figures. As a result, multi-modal, multi-document scientific reasoning remains underexplored and lacks systematic evaluation. To address this gap, we introduce PaperScope, a multi-modal multi-document benchmark designed for agentic deep research. PaperScope presents three advantages: (1) Structured scientific grounding. It is built on a knowledge graph of over 2,000 AI papers spanning three years, providing a structured foundation for research-oriented queries. (2) Semantically dense evidence construction. It integrates semantically related key information nodes and employs optimized random-walk article selector to sample thematically coherent paper sets, thereby ensuring adequate semantic density and task complexity. (3) Multi-task evaluation of scientific reasoning. It contains over 2,000 QA pairs across reasoning, retrieval, summarization, and problem solving, enabling evaluation of multi-step scientific reasoning. Experimental results show that even advanced systems such as OpenAI Deep Research and Tongyi Deep Research achieve limited scores on PaperScope, highlighting the difficulty of long-context retrieval and deep multi-source reasoning. PaperScope thus provides a rigorous benchmark alongside a scalable pipeline for constructing large-scale multi-modal, multi-source deep research datasets.

1 Introduction

PaperScope addresses the limited evaluation of multi-modal, multi-document scientific research by benchmarking retrieval, reasoning, synthesis, and generation across heterogeneous sources. Its graph-guided construction supports this evaluation, while experiments show substantial capability gaps in current agents.

1 Introduction

  • Existing benchmarks and agent evaluations underrepresent multi-modal, multi-document scientific reasoning, especially in large-file settings.Many current systems primarily use internet text and overlook stored documents containing dense visual and symbolic information.Source: §1 · p. 1
  • Existing agents show insufficient accuracy and substantial limitations in multimodal understanding, retrieval, and multi-source information integration on large-scale document research.The evaluation covers standard ReAct-based and Deep Research agents.Source: §1 · p. 2
  • PaperScope evaluates large-scale retrieval, cross-document reasoning, and multi-source synthesis in realistic multi-modal scientific research settings.Its questions draw on heterogeneous sources and modalities.Source: §1 · pp. 1–2
  • The benchmark uses a heterogeneous corpus and knowledge-graph-guided construction to provide fine-grained, near-unique annotations and a reproducible dataset pipeline.Questions are built from document sets sampled by random walks on a knowledge graph, with quality control over documents and annotations.Source: §1 · p. 2

2 Related Work

Prior scientific-document benchmarks and general agent benchmarks leave substantial aspects of multi-document, multi-modal scientific research unevaluated. PaperScope targets these gaps with scientific subsets and tasks spanning retrieval, grounding, reasoning, and solution generation.

2 Related Work

  • Scientific-document benchmarks largely center on single papers or sources, leaving multi-document retrieval, reasoning, summarization, and problem solving insufficiently explored.Some datasets add cross-source or citation-based settings but do not support genuine multi-document synthesis.Source: §2 · p. 2
  • General agent benchmarks emphasize web-based information seeking and provide limited coverage of scientific documents or multi-modal evidence across local files.Their tasks rarely require integrating multiple local files or synthesizing visual, symbolic, and algorithmic information.Source: §2 · p. 2
  • PaperScope addresses these gaps by evaluating cross-document retrieval, multi-modal grounding, and multi-resource reasoning alongside understanding and solution generation.The comparison framework identifies support for understanding, reasoning, retrieval, solution, thinking, and generation.Source: §2 · pp. 2–3, Table 1 · p. 3

3 PaperScope Bench

PaperScope is a multi-modal, multi-document benchmark built from a curated scientific-paper corpus to evaluate research tasks from retrieval through synthesis. Its four meta-categories organize 2,400 questions across 11 sub-tasks.

3.1 Overview

  • 2,400 questions across 11 sub-tasks are organized into Topic Induction, Multi-Document Reasoning, Summary, and Solution categories.The categories are structured to mirror a research workflow from retrieval to synthesis.Source: §3.1 · p. 3

3.2 Data Collection

  • The benchmark corpus is curated from 25,495 ArXiv and OpenReview papers from 2023–2025 across more than 20 AI venues.PDFs are filtered for completeness, readability, and quality.Source: §3.2 · p. 3

3.3 Task Creation

  • Tasks cover implicit and explicit topic retrieval, cross-document multimodal reasoning, multi-paper summaries, and evidence-grounded solution generation.Reasoning includes figure, table, chart, formula, algorithm, and full-paper sub-tasks; summaries cover trends, methods, and fine-grained results.Source: §3.3 · pp. 3–4

3.4 Statistics of Benchmark

  • The benchmark distributes its questions across four major categories, with task and sub-task counts summarized in Table 2.The supplied passage does not provide the individual counts.Source: §3.4 · p. 4, Table 2 · p. 4

4 Bench Construction Methodology

PaperScope constructs multi-document research sets from a knowledge graph of scientific papers and shared information nodes. Its pipeline combines structured extraction, semantic merging, graph-based article selection, and quality-control strategies.

4.1 Overview

  • The construction pipeline extracts key information, builds a graph, merges semantically related nodes, then uses ORWAS to select papers sharing key information.The overview describes these as three stages: graph construction, HNSW-based retrieval and merging, and optimized random-walk selection.Source: §4.1 · pp. 4–5, Figure 2 · p. 5

4.2 Graph Construction

  • Each paper is represented by a title node linked to 13 extracted information types, including methods, results, formulas, algorithms, figures, tables, and limitations.The extraction is performed with a large language model.Source: §4.2 · p. 5

4.3 Semantic Disambiguation and Merging

  • Nodes are embedded in a shared 4096-dimensional space and merged with same-type neighbors above a threshold while preserving provenance.The process uses HNSW retrieval and a coarse-to-fine schedule across information types.Source: §4.3 · p. 5

4.4 Optimized Random-Walk Article Selector (ORWAS)

  • ORWAS samples and ranks paper-set candidates on a graph of about 30,000 nodes and 200,000 edges using coverage, diversity, consistency, and redundancy.Its strategy includes stratified initialization, structure-aware walks, cached adjacency, parallelization, and candidate-size memory caps.Source: §4.4 · p. 5

4.5 Quality Control

  • Quality control derives questions from known facts for answer uniqueness, places key evidence exclusively in visual modalities, and retains questions that strong models fail on one web-search attempt.The difficulty screen names Gemini 2.5-Pro and GPT-5.Source: §4.5 · pp. 5–6

5 Experiments

Experiments evaluate scientific research agents across four task types and find persistent weaknesses in multimodal, multi-document reasoning. Ablations identify reasoning complexity, retrieval quality, and ORWAS settings as relevant factors, while a pilot study tests transfer beyond AI papers.

5.1 Evaluation

  • Evaluation covers Topic Induction, Reasoning, Summary, and Solution, using task-specific retrieval, exact-match, hybrid summary, and GPT-5-judged solution metrics.Summary evaluation combines induction scores with five GPT-5 dimensions; Solution uses Analysis and Technical Scores.Source: §5.1 · p. 6

5.2.1 Experiment Settings

  • Experiments use a stratified 200-QA subset in a multimodal PDF environment, with ReAct agents using FileSearch and FileVisit and baselines spanning MLLM ReAct and Deep Research systems.The environment provides high-resolution images and structured PDF–Markdown content for joint text-image reasoning.Source: §5.2.1 · p. 6, §5.2.2 · p. 6

5.3 Main Results

  • Deep-research agents outperform LLM-based ReAct baselines on the heavily weighted Reasoning task, with this advantage depending critically on stable tool calling.Table 3 reports macro-averaged scores across Reasoning, Induction, Summary, and Solution.Source: §5.3 · p. 6, Table 3 · p. 7
  • ReAct systems achieve strong Summarization results but remain constrained by weak Reasoning and Solution scores.The paper links this gap to limited multi-source synthesis, cross-modal alignment, and structured inference.Source: §5.3 · pp. 6–7
  • Models struggle with figures, tables, and mathematical expressions, while long-context interactions can trigger forgetting and hallucinations that destabilize reasoning.The reported difficulties include broken logical chains, misaligned evidence, and insufficient multimodal fusion.Source: §5.3 · pp. 7–8

5.4 More Ablation Analysis

  • Providing oracle support documents improves performance, while replacing local retrieval with generic WebSearch substantially lowers scores.The authors attribute the WebSearch decline to unreliable retrieval of supporting documents and greater reliance on parametric knowledge.Source: Figure 3 · p. 7, §5.4 · p. 8
  • ORWAS defaults of L = 100, β = 0.3, and W = 10000 maximize combination diversity at low computational cost, while very short or long walks and excessive exploitation reduce effectiveness.W = 10,000 reaches efficiency saturation in roughly 60 seconds; further scaling adds time without significant diversity gains.Source: §5.4 · pp. 8–9, Table 4 · p. 9

5.5 Cross-Domain Generalizability

  • A pilot on Medicine and Mechanics papers produced knowledge graphs with over 690 nodes and 35,000 edges per domain and a 95% expert satisfaction rate.The pipeline linked cross-document entities and modalities without domain-specific heuristic tuning.Source: §5.5 · p. 9

6 Conclusion

PaperScope is a multimodal, multi-document benchmark and construction pipeline built from knowledge graphs spanning over 2,000 AI papers. Its 2,400 questions cover four capabilities, and even state-of-the-art agents face substantial challenges in long-context retrieval and multi-document reasoning.

6 Conclusion

  • PaperScope contains 2,400 questions across Topic Induction, Multi-Document Reasoning, Summary, and Solution, built from knowledge graphs spanning over 2,000 AI papers.The paper reports Grok 4’s score of 40.95 and describes persistent challenges in long-context retrieval and multi-document reasoning.Source: §6 · p. 9

Limitations

The limitations identify the benchmark’s finite local corpus as a boundary on how fully it reflects open-domain document collections and noisy retrieval.

Limitations

  • Because PaperScope uses a finite local corpus, it may not fully reflect challenges from unbounded document collections and noisy retrieval.The authors also note heuristic article selection, possible LLM-generated query and annotation noise, and limited baseline coverage under fixed inference budgets.Source: Limitations · p. 10

A.1 Detailed Benchmark Statistic

The appendix reports corpus and graph statistics, describes ORWAS for sampling thematically coherent paper subsets, and details task construction and annotation checks. Its annotation workflow grounds questions in evidence across documents and modalities, then audits structured tasks for coherence and accuracy.

A.1 Detailed Benchmark Statistic

  • The benchmark statistics include corpus-level figures and the distribution of supporting documents per question, which reflects each task’s multi-document dependency.The appendix points to Tables 5 and 6 for these statistics.Source: §A.1 · p. 12, Table 5 · p. 12, Table 6 · p. 12

A.2 Data Collection

  • The study uses documents under CC 4.0 licenses and excludes personally identifiable or sensitive private author data from its content-analysis queries.Source: §A.2 · p. 12

B.1 Detailed Graph Statistic

  • The appendix describes two heterogeneous semantic graphs and provides their structural statistics and visualizations.It also identifies Table 7 as the summary of graph structural properties and Figure 5 as a visualization.Source: §B.1 · p. 12, Table 7 · p. 12, Figure 5 · p. 13

B.2 Detailed Optimized Random Walk Article Selector (ORWAS)

  • ORWAS uses random walks on a heterogeneous paper–key-information graph to select compact, thematically coherent paper subsets.Its stratified sampling starts 0.7W walks from article nodes and 0.3W from high-frequency key-information nodes.Source: §B.2 · p. 12, §B.1 · p. 12

B.3 Detailed Task Formulation

  • The appendix provides prompts for three synthetic task types and notes that Summary tasks have a dedicated prompt for trend questions.The prompts are shown in Figures 6, 7, and 8.Source: §B.3 · p. 12

B.4.1 Expert Annotation for Cross-Document Reasoning

  • Reasoning questions are derived from irrefutable evidence across documents, with visual evidence and multi-hop inference required and text-only or single-step questions rejected.The inverted-construction workflow is intended to ensure answer uniqueness within a closed retrieval scope.Source: §B.4.1 · p. 13

B.4.2 Expert Verification for Structured Tasks

  • Structured-task audits check topic-cluster exclusivity and coherence, summary alignment of experiments and methods, and logical organization of solution conditions without hallucination.The appendix describes a bifurcated construction strategy: expert annotation for complex reasoning and human-in-the-loop verification for Topic Induction, Summary, and Solution.Source: §B.4.2 · p. 13, §B.4 · p. 13

B.5 Human-Model Evaluation Alignment

The appendix reports alignment between automated and human evaluation, tests the benchmark’s dependence on visual evidence, and documents evaluation prompts and retrieval tools. It also describes the baseline configuration and the benchmark’s verification process.

B.5 Human-Model Evaluation Alignment

  • For Solution, Pearson r = 0.6385; the Analysis dimension reaches 0.7173, and Spearman ρ > 0.6 indicates ranking stability.The reported automated–human correlations are statistically significant at p < 0.0001.Source: §B.5 · p. 14, Table 9 · p. 14

B.6 Multi-modal Dependency and Data Verification

  • Text-only performance drops by an average of 81.7% when visual inputs are removed, and only approximately 8% of questions are solvable from text alone.The ablation uses 100 reasoning tasks and removes charts and figures, retaining OCR-extracted text.Source: Table 10 · p. 14, §B.6 · p. 14
  • Corpus-A verification defines uniqueness as constraint-based determinability within the local corpus; AI filtering removed 9.98% (112 items), followed by expert review removing 7.45% (82 items).The two stages check difficulty, exclusivity, semantic coherence, and structural completeness.Source: §B.6 · p. 14

C More Detailed of baseline

D.1 Evaluation Prompts

  • Summary responses are judged on Fluency, Relevance, Accuracy, Creativity, and Overall Quality, while Solution uses Analysis and Technology scores.GPT-5 adjudicates Summary responses; Solution evaluation receives relevant analysis and technical knowledge plus gold explanations and solutions.Source: §D.1 · p. 15

D.2 Detailed Tools

  • The FileSearchTool combines multimodal embeddings and OCR to index text and images, retrieve top-k candidates, and return image descriptions alongside text results.The system uses Ops-MM-embedding-v1-7B, DeepSeek-OCR, and a FAISS inner-product index.Source: §D.2 · p. 15

D.3 Detailed Ablation Experiments

The appendix describes ablations on the Reasoning subset that separate retrieval from reasoning and test generic web search. It also reports task-specific analyses and uses cases to illustrate both successful and failed agent behavior.

D.3 Detailed Ablation Experiments

  • The retrieval ablations use the difficult, objectively scored Reasoning subset to distinguish retrieval-module contributions from agent reasoning capabilities.The subset was selected for its strict multimodal dependencies and high intrinsic difficulty.Source: §D.3 · p. 15

D.3.1 Impact of Retrieval Necessity: The Oracle Setting

  • The oracle setting bypasses active search by supplying gold-standard parsed documents, comparing performance under ideal context with the full pipeline’s retrieval uncertainty.Source: §D.3.1 · pp. 15–16

D.3.2 Impact of Generic Web Search

  • The generic-search ablation uses Bocha web search to test benchmark vulnerability, with underperformance expected if tasks enforce answer uniqueness and multimodal dependence.The stated rationale is that standard search may not index or synthesize the visual evidence and cross-document inference required.Source: §D.3.2 · p. 16

D.4 Detail Evaluation Results

  • Task-specific analyses examine induction and solution results where models score above zero, with partial all-subtask results reported for a subset of models.The analyses aim to expose patterns obscured by aggregate metrics.Source: §D.4 · p. 16

D.5 Case Study

  • The case study contrasts a Grok-4 success on a full-paper reasoning question with a failure to retrieve all gold documents and understand table and chart results.Source: §D.5 · p. 16

E Error Attribution Analysis

A manual review of Grok-4 failure cases identifies multimodal extraction, retrieval granularity, hallucination, and multi-step reasoning as recurring bottlenecks. Fine-grained visual understanding is the largest reported category.

E Error Attribution Analysis

  • Experts manually reviewed 40 randomly sampled Grok-4 failures against gold-standard documents to attribute error modes.The resulting breakdown is presented in Table 15.Source: E Error Attribution Analysis · p. 16
  • Multimodal information extraction accounts for 37.5% of failures and is identified as the primary weakness, including misreading table structure or chart coordinates despite retrieving the correct paper.Source: E Error Attribution Analysis · p. 16
  • Multi-step reasoning errors account for 22.5%; small deviations in initial metric extraction can compound during cross-paper comparisons and cause final answers to drift.Source: E Error Attribution Analysis · p. 16
  • Retrieval granularity and hallucination account for 35% combined: 20% of errors involve overly broad search scopes, while 15% involve hallucination, often after exact evidence is missed.Source: E Error Attribution Analysis · p. 16

Related papers

Loading 2604.11307v1…