PaperMind: Benchmarking Agentic Reasoning and Critique over Scientific Papers in Multimodal LLMs

• Existing benchmarks typically isolate visual perception, single-paper answer extraction, summarization, or general tool use rather than coordinated scientific analysis.Source: §1 · pp. 1–2
• The benchmark requires models to ground figures in context, interpret experiments, retrieve and integrate external evidence, and assess claims in realistic workflows.Source: §1 · pp. 1–2
• PaperMind evaluates integrated reasoning across multimodal grounding, experimental interpretation, evidence synthesis with tool use, and critical assessment.Source: §1 · p. 2
• Its agentic tasks include tool-augmented evidence retrieval and rebuttal-style critique grounded in real peer-review discussions.Source: §1 · p. 2
• Evaluations across open-source and closed-source multimodal LLMs provide systematic comparisons that reveal performance gaps and characteristic scientific-reasoning failure modes.Source: §1 · p. 2

2.1 Scientific Paper QA

• Existing scientific QA benchmarks cover single-document comprehension, factual retrieval, multimodal figures and tables, multi-document reasoning, and citation-based retrieval, but usually assess isolated capabilities.Source: §2.1 · p. 3
• PaperMind jointly evaluates diverse scientific QA paradigms to assess comprehensive understanding rather than narrowly defined question-answering skills.Source: §2.1 · p. 3

2.2 Tool-Augmented Agentic Reasoning

• Prior agentic systems use reasoning interleaved with actions, program execution, external APIs, modular solvers, or multi-agent collaboration, with benchmarks such as ToolHop and PaperArena extending these capabilities.Source: §2.2 · p. 3
• PaperMind adds fine-grained analysis of tool-use failure patterns and evaluates models’ ability to rebut reviewer-style critiques beyond task-level success metrics.Source: §2.2 · p. 4

3.1 Paper Collection

• The benchmark draws on 3,000 open-access papers across agriculture, biology, chemistry, computer science, medicine, physics, and economics, plus OpenReview discussions for Critical Assessment.Source: §3.1 · p. 4

3.2 Filtering Process

• Filtering removes papers shorter than five pages, longer than twenty pages, or affected by severe formatting problems before structured extraction and deduplication.Source: §3.2 · p. 4

3.3 QA-pair Construction

• The four task families are grounded in figures, tables, references, and peer-review discussions, with domain-wise question distributions documented in Figure 2.Source: §3.3 · p. 4
• Multimodal Ground asks models to caption figures using introductions as context, while Experimental Interpretation asks for analytical narratives grounded in visual or tabular results and referenced discussion paragraphs.Source: §3.3 · p. 4
• Cross-Source Evidence Reasoning requires autonomous retrieval from the source paper, cited works, or external resources, followed by multi-source evidence synthesis.Source: §3.3 · p. 5
• Critical Assessment retains reviewer questions with substantive author responses and explicit score improvements, while removing trivial or non-informative review content.Source: §3.3 · p. 5

4.2 Main Results

• Gemini 2.5 Pro achieves the strongest Multimodal Ground performance, while Cross-Source Evidence Reasoning shows larger gaps between closed- and open-source models.For Experimental Interpretation, Qwen3-VL-4B-Instruct sometimes has higher F1, but Claude 3.5 Sonnet consistently scores better under LLM-as-a-Judge; Gemini 2.5 Pro leads Critical Assessment.Source: §4.2 · p. 6

4.3.1 Effect of Background Introduction on Scientific Tasks

• Adding introduction-level context raises Multimodal Ground F1 by approximately 13.7% for Gemini 2.5 Pro and 14.5% for Qwen3-VL-4B-Instruct.Qwen’s LLM-as-a-Judge score also improves by around 8.8%, whereas Gemini changes only marginally.Source: §4.3.1 · p. 6

4.3.2 Tool Usage Analysis

• Models rely more on general web search than arxiv_retriever, favoring broad retrieval coverage over strictly targeted academic sources.This analysis concerns tool use in Cross-Source Evidence Reasoning and Critical Assessment.Source: §4.3.2 · p. 6
• Increasing the maximum tool budget from 4 to 6 produces the largest gains, while 8 yields marginal gains and 10 slightly degrades performance.The reported degradation is attributed to longer contexts that can dilute relevant evidence and burden reasoning and generation.Source: §4.3.2 · pp. 6–7, Figure 4 · p. 7
• Explicit source information improves Gemini-2.5-Pro by 22.6% in F1 and 25.4% in LLM-as-a-Judge, showing that independently retrieving sources adds substantial difficulty.The gains suggest that source identifiers reduce retrieval burden and let models focus on evidence integration and answer synthesis.Source: §4.3.2 · p. 7, Table 5 · p. 7

4.3.3 Failure Mode Analysis

• For Qwen3-VL-4B-Instruct, Insufficient Interaction Steps is the most frequent Critical Assessment error at 77%, followed by shallow evidence integration.The model averages 2.27 interaction steps per question and may answer flawed reviewer premises directly instead of rebutting them.Source: §4.3.3 · pp. 7–8, Figure 6 · p. 8

4.4 A Case Study

• In the case study, Gemini forms an explicit plan and gathers additional arXiv and web evidence, whereas Qwen proceeds directly to tools and relies on document-level information.Both models use PDF parsing and contextual retrieval, but Gemini’s broader tool use is associated with more deliberate reasoning in this example.Source: §4.4 · p. 8, Figure 7 · p. 8

• PaperMind assesses Multimodal Ground, Experimental Interpretation, Cross-Source Evidence Reasoning, and Critical Assessment as interdependent scientific reasoning tasks.The benchmark targets higher-level reasoning required in scientific research workflows beyond isolated retrieval and summarization.Source: §5 · p. 8
• Evaluations of open- and closed-source LLMs uncover substantial performance disparities and recurring failure modes.The benchmark is positioned as a basis for more systematic evaluation and development of agentic scientific-literature systems.Source: §5 · p. 8

• Exclusive reliance on LLM-as-a-Judge leaves the consistency, stability, and human-preference alignment of judgments across question types and domains unresolved.The authors identify evaluation stability and granularity as areas for improvement.Source: Limitations · p. 9

• Additional details are provided on the paper-dataset statistics supporting benchmark construction.The section introduces these statistics before analyzing page-length distributions.Source: A Benchmark Construction · p. 12
• The dataset analysis reports source-paper page-length distributions to characterize the diversity and scale of documents used to construct task instances.The analysis is presented in Table 6.Source: A Benchmark Construction · p. 12
• Representative task examples are presented with the tool libraries required to solve them.These examples are shown in Figure 8.Source: A Benchmark Construction · p. 12

• F1 measures consistency between model outputs and reference answers, while GPT-4o assigns each prediction a five-level rating using a rule-based script.The evaluation reports domain-wise averages and variance for both metrics.Source: B Evaluation Details · p. 12
• LLM-as-a-Judge accuracy is computed as the proportion of predictions receiving a score of at least 4.Source: B Evaluation Details · p. 12

C Additional Experiments

• The report gives LLM-as-a-judge F1 and accuracy variance across four tasks, and analyzes introduction context, interaction depth, tool-call limits, and an eight-way error taxonomy.The analyses use Tables 7–12 and 14, while the error taxonomy appears in Figure 13.Source: C Additional Experiments · p. 12
• The eight-way error taxonomy characterizes failures including question misunderstanding, reasoning-plan failure, incorrect tool use, insufficient interaction, inaccurate queries, shallow evidence integration, generic assumptions, and missed question errors.Source: C Additional Experiments · p. 12

D.1 Personal Information Usage

• Question-answer pairs are built solely from open-access scholarly materials and filtered for inaccurate, misleading, or inappropriate content.The stated purpose is research and benchmarking, with no data privacy concerns identified from this source restriction.Source: §D.1 · p. 13

D.2 Human Annotation

• Human involvement was limited to author-performed quality verification for Task 3, checking factual consistency and evidence alignment without collecting personal information.Source: §D.2 · p. 13

• Benchmark performance may be overinterpreted as real-world research competence and may propagate biases from the source literature or pretrained models.The authors mitigate these risks by documenting data sources, task scope, and evaluation limitations.Source: E Potential Risks · p. 13

• LLMs assist with candidate question and answer generation, but authors manually validate factual correctness, evidence alignment, and task relevance.Source: F Use of AI Assistants · p. 13
• LLMs also serve as an automatic evaluation component and limited language-editing aid, while authors retain responsibility for task design, analysis, and conclusions.Source: F Use of AI Assistants · p. 13
• The cited materials cover dataset statistics, four-task design, LLM-as-a-judge variance and accuracy on Multimodal Ground, and prompt settings for three task configurations.Judge scores use a 5-point scale, with accuracy treating scores above 4 as success.Source: Table 6 · p. 14, Figure 8 · p. 14, Table 7 · p. 14
• LLM-as-a-Judge analyses use five-point scores, with variance and accuracy reported for task-specific evaluations.Source: Table 8 · p. 16, Table 9 · p. 16, Table 14 · p. 18
• Supplementary experiments examine maximum tool-call limits for Cross-Source Evidence Reasoning and Critical Assessment.Source: Table 11 · p. 16, Table 12 · p. 17, Table 15 · p. 18