PaperlayerView exact revision on arXiv

Source-linked AI summary

Sci-MMR: Benchmarking Multi-Step Evidence-Grounded Scientific Reasoning in Multimodal Agents

Jiaqiang Li, Yajie Yang, Zhiheng Xi, Jiadong Chen, Enyu Zhou, Senjie Jin, Yang Nan, Jiazheng Zhang, Han Wang, Yanxin Li, Dingwei Zhu, Bicheng Deng, Yuhui Wang, Xiang Zheng, Qi Zhang, Lei Bai, Xingjun Ma, Tao Gui

arXiv:2609.11243v1cs.AI

TL;DR

Existing multimodal benchmarks emphasize final-answer accuracy rather than traceable evidence support. Sci-MMR evaluates multi-step evidence-grounded scientific reasoning with structured argument graphs and finds persistent bottlenecks in evidence acquisition and integration, showing that answer accuracy alone can overestimate reliability.

  • Problem

    Existing multimodal benchmarks largely evaluate final-answer accuracy, leaving whether predictions are supported by traceable scientific evidence unresolved.

  • Method

    Sci-MMR represents 235 multi-hop tasks across four disciplines and 35 domains as argument graphs linking claims, citation-grounded knowledge, visual evidence, and supporting regions.

  • Results

    Across eight frontier multimodal models, evidence acquisition accounts for 57.2% of Direct Visual Reasoning errors, while evidence integration accounts for 31.8% of failures.

  • Takeaways & Limitations

    Answer accuracy alone can overestimate scientific reasoning reliability, motivating evaluation that traces the complete structured evidence behind conclusions.

  • Takeaways & Limitations

    Argument graphs represent one plausible reconstruction rather than unique ground truth, and coverage metrics evaluate explicit final responses rather than internal reasoning.

Abstract

from arXiv · show

Autonomous research agents are increasingly expected to search the literature, analyze experimental evidence, and generate scientific hypotheses. These capabilities require multi-step evidence grounded reasoning that progressively acquires, integrates, and verifies evidence before reaching a conclusion. Existing multimodal benchmarks, however, largely evaluate final-answer accuracy, leaving open whether predictions are actually supported by traceable scientific evidence. We introduce Sci-MMR, a benchmark for multi-step evidence-grounded scientific reasoning built on structured argument graphs linking scientific claims, citation-grounded knowledge, visual evidence, and supporting regions. Sci-MMR comprises 235 multi-hop reasoning tasks spanning four scientific disciplines, with an average of nine figure panels per task. Evaluating eight frontier multimodal models, we find that answer accuracy consistently exceeds complete-evidence recovery rate by more than 20%, revealing a substantial gap that answer-only evaluation is structurally unable to capture. Through controlled interventions, we identify two fundamental bottlenecks. First, evidence acquisition: models struggle to extract complete structured evidence from scientific figures, accounting for 57.2% of failures. While cropping tools yield modest gains (+4.5 points), providing gold evidence improves accuracy by up to 37.0 points, indicating difficulty in assembling complete multi-region evidence. Second, evidence integration: models struggle to translate available evidence into correct conclusions, accounting for 31.8% of failures, while even with gold evidence the strongest model achieves only 69.1% accuracy on the hardest tasks. These findings indicate that current answer-centric benchmarks substantially overestimate the evidence-grounded reasoning capabilities of multimodal research agents

1. Introduction

Sci-MMR evaluates whether multimodal agents can acquire, organize, integrate, and verify distributed scientific evidence rather than merely produce correct answers. Its benchmark and controlled interventions expose substantial bottlenecks in complete evidence acquisition and evidence integration.

  • Benchmark and motivation: Existing multimodal evaluations mainly measure final-answer correctness, leaving models’ ability to acquire and organize justificatory evidence unresolved.
  • Benchmark and motivation: Sci-MMR represents 235 multi-hop tasks across four disciplines and 35 domains as argument graphs linking claims, citations, visual evidence, and supporting regions.Each task aggregates evidence across an average of nine figure panels.
  • Evidence acquisition: 57.2% of Direct Visual Reasoning errors arise from evidence access and grounding, with models recovering complete evidence in only 12.7% of runs.Models find at least one relevant region in 62.1% of runs, indicating that assembling complete multi-region evidence is the central acquisition difficulty.
  • Evidence acquisition: Visual cropping improves accuracy by 4.5 points, whereas supplying gold evidence statements yields 19.1–37.0% gains, showing that complete localization remains difficult.
  • Evidence integration: 31.8% of failures reflect evidence integration errors, and the strongest model reaches only 69.1% accuracy on the hardest tasks even with gold evidence.Evidence-grounded reasoning therefore remains difficult after the required evidence is available.

2. Related Work

Prior scientific multimodal benchmarks generally evaluate final answers or reasoning within limited settings, rather than reconstructing the evidence supporting scientific claims. Sci-MMR extends this line of work by explicitly representing and evaluating evidence dependencies.

  • Prior benchmarks: Existing scientific multimodal benchmarks typically assess figure-grounded question answering, claim verification, or paper understanding through final-answer recovery.
  • Beyond answer-centric evaluation: Recent benchmarks add intermediate reasoning, agent behavior, and evidence-fidelity checks, but do not replace answer-centric evaluation with direct evidence-dependency reconstruction.

3. Sci-MMR and Evaluation

Sci-MMR models scientific reasoning as hierarchical evidence dependencies and evaluates whether models recover and integrate the complete reasoning chain. Its benchmark construction combines automated graph generation and filtering with domain-expert verification across diverse multi-hop tasks.

  • 3.1. Argument Graph Formulation: Sci-MMR represents scientific arguments as a directed acyclic graph linking visual sources, evidence, knowledge, and hierarchical claims through support relationships.Task solving requires recovering relevant evidence and knowledge nodes and integrating them to infer a unique root claim.
  • 3.2. Benchmark Construction and Composition: The benchmark uses automated extraction and graph construction, while domain experts localize evidence to visual regions, rewrite observations conservatively, and verify the complete argument graph.Experts provide the final scientific check after automated processing.
  • 3.2. Benchmark Construction and Composition: Tasks are generated by sampling verified argument subgraphs, converting their required observations, intermediate claims, and knowledge into multi-hop questions with grounded reference answers.Automated filters remove shallow or low-quality candidates before final manual review.
  • 3.2. Benchmark Construction and Composition: Sci-MMR contains 235 multi-hop tasks spanning four disciplines and 35 domains, with an average of nine figure panels per task.Figure 3 summarizes the benchmark’s discipline–domain distribution and task-complexity distributions.
  • 3.3. Evaluation Metrics: Evaluation measures Answer Accuracy, Evidence Coverage, and Claim Coverage to distinguish final conclusions from evidence acquisition and evidence integration.Evidence Coverage assesses recovery of grounded observations, while Claim Coverage assesses composition of recovered evidence into required intermediate claims.

4. Experiments

Experiments evaluate eight multimodal models across progressively constrained settings and diagnose evidence-grounded reasoning failures. Results show that models often recover incomplete evidence, benefit from evidence hints, and still struggle with evidence integration and localization.

  • Evaluation Design: The experiments use four progressively constrained settings to isolate evidence acquisition, evidence integration, and final-answer reasoning.The settings include Caption-only, Direct Visual Reasoning, Evidence-Hint Reasoning, and Agentic Tool-Use.
  • Main Results: GPT-5.5 reaches 67.7% accuracy in Direct Visual Reasoning and 69.8% with Agentic Tool-Use, while tool-assisted gains remain modest.Claude Opus 4.8 gains the most from tools at 5.1 points; GPT-5.5 remains strongest overall.
  • Evaluation Design: Evidence coverage does not necessarily translate into correct scientific conclusions, separating evidence recovery from successful reasoning.The evaluation therefore distinguishes answer accuracy, evidence coverage, and claim coverage across multiple settings.
  • RQ1: Evidence Availability: 27.4 points: gold evidence raises mean accuracy from 46.1% to 73.5%, but hard-task accuracy reaches only 54.8%.Hard tasks gain 44.1 points, compared with 27.2 points on medium tasks and 5.8 points on easy tasks.
  • RQ2: Evidence Acquisition: 12.7% of crop-called runs achieve complete required-region coverage, with mean coverage of 34.1% and 37.9% hitting no required region.Among initially incorrect, low-coverage runs, answer rescue rises to 38.7% with at least four annotated-region hits.
  • RQ3: Failure Analysis: 57.2% of incorrect outputs are evidence-related failures, while reasoning-related failures account for 31.8%.Access and localization failures comprise 40.1% of errors, and interpretation failures comprise 27.5%.

5. Limitations

Sci-MMR’s argument graphs represent plausible reconstructions rather than unique ground truth, and evaluation focuses on expressed responses rather than internal reasoning. Expert verification and cross-judge agreement help, but construction subjectivity and LLM-evaluation bias remain.

  • Argument graphs encode one plausible reconstruction of a paper’s evidential structure, so alternative valid granularities, claims, and support relations may exist.The authors note that expert verification does not eliminate construction subjectivity.
  • Evidence Coverage and Claim Coverage assess evidence and claims expressed in final responses rather than models’ internal reasoning processes.
  • LLM-based assessment may remain sensitive to semantic ambiguity, response style, and evaluator bias despite strong cross-judge agreement.

6. Conclusion

Sci-MMR evaluates multi-step evidence-grounded reasoning through argument graphs that connect visual sources, citation-grounded knowledge, evidence, and claims. Across eight frontier models, gold evidence helps substantially, but errors persist, showing that answer accuracy alone can overestimate reliability.

  • Sci-MMR benchmarks 235 tasks whose argument graphs connect visual sources, evidence, citation-grounded knowledge, and claims.This supports evaluation beyond answer accuracy alone.
  • Supplying gold evidence substantially improves performance, whereas visual cropping tools yield only modest gains across eight frontier multimodal models.
  • Persistent errors after evidence provision reveal challenges in both evidence acquisition and evidence integration, so answer accuracy alone can overestimate reasoning reliability.
  • A proposed next step is extending within-paper argument graphs to cross-paper modeling of complementary or conflicting evidence and claims.

A. Benchmark Composition and Audit

The benchmark contains 235 questions across biology, chemistry, computer science, and physics, with source-pool composition, structural demands, graph audits, and expert verification documented.

  • Sci-MMR contains 235 questions spanning biology, chemistry, computer science, and physics.
  • The composition analysis reports source-pool and benchmark structure before characterizing question-level structural and visual demands.
  • The benchmark documentation includes an audited graph example and expert verification.

A.1. Temporal, Disciplinary, and Domain Coverage

The benchmark’s 235 tasks cover 35 domains across four disciplines and combine nontrivial dependency structures with concentrated local visual evidence. An audited example preserves traceability from claims through evidence nodes to source panels, while construction uses staged validation and rewriting.

  • Temporal, Disciplinary, and Domain Coverage: Physics contributes 88 questions, computer science 63, biology 50, and chemistry 34, while the largest primary domain contains 23 questions.The distribution maintains broad topical coverage while emphasizing physics and computer science.
  • Structural and Visual Complexity: The final benchmark’s 235 tasks have dependency depths of 3, 4, 5, and 6 in counts of 125, 84, 25, and 1, respectively.Graph medians are seven nodes and eight edges, with a minimum required dependency depth of 3.
  • Structural and Visual Complexity: The benchmark’s visual load has a median of five annotated subfigures per question, a mean of 8.97, and a maximum of 104.
  • Audited Argument-Graph Example: In the audited example, 24 nodes and 32 directed relations at depth five connect one final claim and six intermediate claims to 13 evidence nodes grounded in four source figures.Evidence values and source-panel identifiers are retained, and evidence-specific crops preserve legibility and graph-to-panel mapping.
  • Construction and Quality Control: Deterministic validation removes 54% of candidate tasks and rubric assessment filters a further 17% before conditional rewrites and re-evaluation.The reported rewrite flags are overlapping events rather than sequential rates.
  • Benchmark Composition and Audit: Table 4 organizes all 35 primary domains within discipline totals, while Table 3 documents task construction and quality control.

A.4. Expert Verification Protocol

Sci-MMR tasks are constructed and verified through expert review of questions, evidence regions, claims, and support relations. The protocol combines structural validation, rubric-based screening, independent review, adjudication, and graph-level checks.

  • Expert review: Each task is independently checked by a discipline-matched expert and a senior cross-reviewer, with an additional senior reviewer used when needed.Reviewers inspect the question, supporting regions, evidence statements, claims, and graph relations.
  • Validation criteria: The protocol checks evidence entities, conditions, directions, comparisons, values, units, provenance, support relations, connectivity, and acyclicity.Unsupported inference and answer leakage are explicitly rejected during evidence and graph review.
  • Evidence grounding: Evidence labels map graph nodes to source figures and panels, while crops preserve the original source content for traceable visual grounding.The audited example contains seven claim nodes, 13 evidence nodes, four source figures, and 32 support relations.
  • Expert review: 235 retained tasks were selected from 260 reviewed tasks after expert construction-stage screening, while 25 tasks were rejected.Among retained tasks, 145 were directly accepted, 69 required minor revision, and 21 required major revision or redrawing; 24 required third-reviewer adjudication.
  • Validation criteria: Question inclusion uses deterministic validation followed by an eight-dimensional language-model rubric with structural, dependency, anchor-coverage, and non-leakage checks.The overall passing threshold is 85, while normalized scores below 0.5 on any critical dimension trigger revision.

C.3. Human and Cross-Judge Audit

The benchmark’s human and cross-judge audits use balanced sampling, independent annotation, and senior adjudication to validate correctness, evidence coverage, and claim coverage. The audit also compares caption-only and visual reasoning performance.

  • Audit design: The audit samples 384 responses evenly across 8 models, 4 settings, and 3 fixed difficulty strata, with each response assigned to three annotators.A fifth senior adjudicator resolves disputed atomic labels after consulting the source paper and argument graph when necessary.
  • Audit validation: Three-annotator unanimous agreement reaches 83% for evidence-node decisions and 86% for intermediate-claim decisions, while Gemini reaches 85% and 87% agreement with adjudicated consensus.For final Accuracy, Gemini reaches 92% exact agreement with adjudicated human consensus.
  • Cross-judge consistency: Mean exact agreement is 90.8% for correctness atoms, 83.3% for evidence coverage, and 87.0% for claim coverage across shared audit records.Pairwise agreement between Gemini 3.5 Flash and peer judges ranges from 76.3% to 88.2% for evidence-node labels.
  • Caption-only diagnostic: Caption-only accuracy is 38.7%, compared with 46.1% under Direct Visual Reasoning, with caption-only E-Cov. and C-Cov. of 26.5% and 26.8%.Captions provide useful semantic cues for some tasks but do not replace the original visuals overall.
  • Discipline comparison: Direct Visual Reasoning accuracy is highest for chemistry at 56.6% and lowest for physics at 41.6% in the analysis set.The discipline comparison is descriptive; the main results use fixed graph-annotation difficulty strata.

D.3. Coverage and Correctness

Under Direct Visual Reasoning, higher explicit evidence coverage is consistently associated with higher correctness. The benchmark therefore distinguishes whether models state supporting evidence from whether they reach the correct final answer.

  • Coverage diagnostic: Figure 11 compares pooled accuracy and C-Cov. across E-Cov. bins and contrasts each model’s accuracy at low and high coverage.Low coverage is defined as E-Cov. < 0.3 and high coverage as E-Cov. ≥0.8 in the figure diagnostic.
  • Coverage–correctness relationship: 25.1% accuracy occurs below E-Cov. 0.2 versus 62.6% at E-Cov.≥0.8, a 37.5-point difference across 1,040 observations.The low–high comparison holds within every model and is descriptive rather than causal.
  • Coverage–correctness relationship: MiniMax-M3 achieves the highest model-level E-Cov. at 61.5% and C-Cov. at 51.7%, but its accuracy is 45.5%.The result illustrates that expressing the intended support chain does not guarantee correct integration into the final answer.

E.1. Evidence Hint Gains

Evidence hints produce large accuracy gains, whereas agentic cropping yields smaller and less uniform improvements. The paired results show that explicit evidence access helps, but output coverage and correctness can still diverge.

  • Evidence Hint Gains: Evidence-Hint Reasoning rescues 597 initially wrong answers and loses 82 initially correct answers, for a net gain of 515 correct observations (+27.4 points).Every model has substantially more rescues than losses, while C-Cov. increases for all eight models by 15.8–33.1 points.
  • Agentic Tool-Use Changes: Agentic Tool-Use yields 260 rescues and 175 losses, for a net gain of 85 correct observations (+4.5 points), smaller and less uniform than Evidence Hint.Intern-S2 improves accuracy from 26.8% to 34.0% while its E-Cov. falls from 41.8% to 32.6% and C-Cov. from 31.2% to 22.1%.
  • Crop access: The answer-rescue rate rises from 20.0% with no annotated-region hit to 38.7% with at least four hits.Greater annotated-region access is associated with stronger evidence recovery and answer rescue, while hit count measures geometric access rather than expressed evidence.
  • Failure composition: Access/localization is the largest individual subtype for every model, with 407 such errors among 1,014 incorrect Direct Visual Reasoning outputs.The eight-way routing categories sum exactly to 1,014 observations.
  • Failure composition: Evidence-related failures comprise 57.2% of errors, while downstream reasoning groups account for 31.8% and audit/other cases for 11.0%.The broader error partition includes 40.1% access/localization, 14.0% Evidence-Hint-not-reflected, 3.1% visual-extraction, 27.5% interpretation residuals, and 4.2% final-answer residuals.

F.5. Representative Error-Pattern Cards

The audited cards distinguish incomplete access, incomplete evidence uptake, partial extraction, downstream interpretation, final selection, and metric-boundary cases across representative errors.

  • Audit-card structure: The cards use a common order—cross-setting outcomes, evidence views, model–gold contrast, and routing implication—to instantiate subtype rules.Subtype counts and shares are defined over 1,014 Direct Visual Reasoning errors.
  • Access and Evidence Hint uptake: Case 1 separates incomplete visual access from incomplete uptake of supplied evidence, while Case 2 reflects only 50.0% of the Evidence Hint.The access case reaches 4 of 14 annotated regions; the uptake case omits output-change and guard-routing constraints.
  • Extraction after complete localization: After complete localization, 14.3% and 33.3% evidence coverage distinguish low from partial visual extraction despite incorrect Agentic Tool-Use outputs.Both cases hit every annotated region, separating geometric access from recovery of scientific meaning.
  • Downstream interpretation and final selection: Complete evidence coverage can still yield errors: Case 5 has zero claim coverage, whereas Case 6 selects the wrong fault plane after covering evidence and claims.These cases separate missing claim expression from incorrect final-answer selection.
  • Cross-setting and localization-proxy audits: Boundary audits include a cross-setting reversal and a correct answer with zero strict IoU@0.5 hits but semantically useful crops.The latter reaches maximum IoU 0.395 and area recall 0.533, preserving the distinction between strict overlap and useful visual access.
Loading 2609.11243v1…