PaperlayerView exact revision on arXiv

Source-linked AI summary

SciVideoBench: Benchmarking Scientific Video Reasoning in Large Multimodal Models

Andong Deng, Taojiannan Yang, Shoubin Yu, Lincoln Spencer, Mohit Bansal, Chen Chen, Serena Yeung-Levy, Xiaohan Wang

arXiv:2510.08559v1cs.CVcs.AI

TL;DR

Existing video benchmarks often target general or college-level scenarios, leaving advanced scientific video reasoning and precise visual grounding under-tested. SciVideoBench introduces research-level experimental-video questions across scientific subjects, built through a semi-automatic process. Evaluations report consistently low accuracy across proprietary and open-source LMMs, while chain-of-thought prompting yields performance gains.

  • Problem

    Many existing video benchmarks focus on general or college-level scenarios and do not adequately test expert knowledge, intricate reasoning, and precise visual perception in advanced scientific experiments.

    Source: §1 · pp. 1–2

  • Method

    SciVideoBench provides 1,000 multiple-choice questions from research-level experimental videos across scientific subjects, generated and refined through an agent-human pipeline with expert validation.

    Source: §1 · pp. 2–3

  • Results

    Proprietary and open-source LMMs show consistently low accuracy on SciVideoBench, and chain-of-thought prompting produces clear performance gains.

    Source: §1 · p. 3, §5 · p. 18

  • Takeaways & Limitations

    SciVideoBench provides a rigorous testbed for evaluating current video-reasoning abilities and points to substantial opportunity for advancing research-level reasoning.

    Source: §1 · p. 3

Abstract

from arXiv · show

Large Multimodal Models (LMMs) have achieved remarkable progress across various capabilities; however, complex video reasoning in the scientific domain remains a significant and challenging frontier. Current video benchmarks predominantly target general scenarios where perception/recognition is heavily relied on, while with relatively simple reasoning tasks, leading to saturation and thus failing to effectively evaluate advanced multimodal cognitive skills. To address this critical gap, we introduce SciVideoBench, a rigorous benchmark specifically designed to assess advanced video reasoning in scientific contexts. SciVideoBench consists of 1,000 carefully crafted multiple-choice questions derived from cutting-edge scientific experimental videos spanning over 25 specialized academic subjects and verified by a semi-automatic system. Each question demands sophisticated domain-specific knowledge, precise spatiotemporal perception, and intricate logical reasoning, effectively challenging models' higher-order cognitive abilities. Our evaluation highlights significant performance deficits in state-of-the-art proprietary and open-source LMMs, including Gemini 2.5 Pro and Qwen2.5-VL, indicating substantial room for advancement in video reasoning capabilities. Detailed analyses of critical factors such as reasoning complexity and visual grounding provide valuable insights and clear direction for future developments in LMMs, driving the evolution of truly capable multimodal AI co-scientists. We hope SciVideoBench could fit the interests of the community and help to push the boundary of cutting-edge AI for border science.

1. Introduction

Existing video benchmarks often focus on general or college-level scenarios, leaving advanced research-level scientific reasoning insufficiently evaluated. SciVideoBench targets this gap with questions grounded in experimental videos, and evaluations find consistently low accuracy among proprietary and open-source LMMs.

1. Introduction

  • Existing benchmarks often emphasize general scenarios, and leading models have saturated on some popular sets while scientific benchmarks can permit answers from memorized knowledge or visually irrelevant assumptions.The paper cites Gemini 2.5 Pro exceeding 85% on VideoMME and Neptune, and reports that some Video-MMMU and MMVU questions can be solved without relying on video evidence.Source: §1 · pp. 1–2
  • SciVideoBench uses research-level experimental videos across four scientific domains and more than 25 subjects, with conceptual, hypothetical, and quantitative questions requiring spatiotemporal grounding, domain knowledge, and logical reasoning.The benchmark comprises 1,000 multiple-choice questions.Source: §1 · pp. 2–3
  • A multi-stage agent-human pipeline mines experimental manuscripts, generates questions with LMMs, and has domain experts validate answers while filtering unanswerable or video-irrelevant questions.The figure emphasizes the combined demands of accurate visual perception, expert knowledge, and logical reasoning.Source: Figure 1 · p. 2, §1 · p. 3
  • Proprietary and open-source LMMs achieve consistently low accuracy on SciVideoBench, exposing substantial room to advance research-level video reasoning.The evaluation also identifies model architecture, reasoning capacity, and perceptual grounding as factors shaping performance.Source: §1 · p. 3

2. Dataset Construction

SciVideoBench combines research-grade experimental videos with a human-reviewed, multi-agent annotation process to create a broad benchmark of scientific video reasoning. Its questions cover three reasoning types and are designed to require detailed experimental understanding.

2.1. Video Collection and Processing

  • The 241 videos come from JoVE and cover four domains and more than 25 scientific subjects, with observable variables, causal relationships, outcomes, and experimental sequences selected to support reasoning.Videos are professionally produced and narratively structured; their manuscript and synchronized narration provide procedural, theoretical, and experimental context alongside the visuals.Source: §2.1 · pp. 3–4, §2.3 · p. 5

2.2. Annotation Pipeline

  • A semi-automatic pipeline combines video, aligned transcripts, and paired papers; manually annotated expert examples guide its multi-agent question generation and refinement.The agents evaluate answerability, check visual grounding and timestamps, and refine questions and options; human experts verify visual evidence and review the final QA pairs.Source: §2.2 · p. 5

2.3. Statistics

  • Videos average 484 seconds, providing a long temporal scale for scientific experiments and the extended reasoning they often require.Figure 4(a) presents the duration distribution.Source: §2.3 · p. 6
  • 1,000 questions are designed to demand research-level knowledge and multi-step reasoning, with conceptual, hypothetical, and quantitative types spanning the benchmark.The question types comprise 245 quantitative, 385 hypothetical, and 370 conceptual questions.Source: §2.3 · pp. 6–7
  • Question lengths indicate greater linguistic complexity than in existing benchmarks, while comparable average answer and distractor lengths reduce annotation bias.Figure 4(b) and (c) show the question- and option-length distributions.Source: §2.3 · p. 6, Figure 4 · p. 6
  • Quantitative questions require video-derived numerical information, hypothetical questions probe experimental operations and controls, and conceptual questions address mechanisms, protocols, and principles inferred from visual cues.The three categories comprise 245 quantitative, 385 hypothetical, and 370 conceptual questions.Source: §2.3 · p. 7

3. Experiments

SciVideoBench evaluations show a large gap between proprietary and open-source models, with quantitative reasoning especially difficult. Human and text-only baselines are also low, while chain-of-thought prompting helps unevenly across model types and reasoning categories.

3.1. Models

  • The evaluation includes five proprietary models and 30 open-source models, spanning open-source parameter sizes from 0.5B to 78B.The tested proprietary models include Gemini-1.5-Pro, Gemini-2.0-Flash, Gemini-2.5-Pro, and GPT-4o.Source: §3.1 · p. 8

3.2. Evaluation Setting

  • Open-source models use model-specific frame sampling, while evaluation temperature is fixed at 0 for stable, reproducible results.The reported sampling counts range from 16 frames for InternVL2 to 768 for Qwen-2.5-VL.Source: §3.2 · p. 8

3.3. Human Evaluation

  • Three graduate students per discipline achieve 17.4% overall accuracy and 14.29% on quantitative reasoning in a closed-book setting.The authors report that the benchmark requires research-level expertise.Source: §3.3 · p. 8

3.4. Blind Baseline Results

  • With video removed, GPT-4o scores 15.80% overall and 11.84% on quantitative reasoning, while Qwen2.5 variants reach at most 18.90%.Both model families remain below 20% without visual input.Source: §3.4 · p. 8

3.5. Proprietary Models vs. Open-Source Models

  • Gemini-2.5-Pro achieves 64.30% overall accuracy, versus 38.80% for the best open-source model, InternVL-3-78B-Instruct; their quantitative scores are 50.61% and 22.04%, respectively.Across all models, proprietary systems average 38.83% overall accuracy versus 18.14% for open-source models.Source: §3.5 · pp. 8, 10, Table 2 · p. 9

3.6. Results of Different Question Types

  • Quantitative reasoning is the most challenging question type across models, while conceptual reasoning generally scores highest, slightly ahead of hypothetical reasoning.The authors relate the quantitative difficulty to precise numerical reasoning and multi-step calculation.Source: §3.6 · p. 10

3.7. Results of Different Disciplines

  • Across disciplines, performance trends are relatively consistent, but Gemini-2.5-Pro performs best in Medicine and worst in Chemistry, opposite Gemini-1.5-Pro’s pattern.The contrast suggests that model architecture and scale can have different effects across domains.Source: §3.7 · p. 10

3.8. The Impact of Chain-of-Thought Prompting

  • Chain-of-thought prompting improves quantitative reasoning across model types, but for most open-source models it hurts overall performance and can worsen conceptual or hypothetical answers.GPT-4o’s quantitative accuracy rises from 11.84% to 34.29%; Qwen2.5-VL-32B-Instruct’s rises from 11.43% to 17.96% even as its overall performance drops 0.7%.Source: §3.8 · pp. 10–12

3.9. The Impact of LMM Model Size

Model size is associated with better performance in some series, but gains are neither linear nor guaranteed; backbone scaling is less effective for quantitative reasoning. Audio adds only modest gains, and error analysis identifies perception, reasoning, and domain-knowledge failures.

3.9. The Impact of LMM Model Size

  • InternVL-3 overall accuracy rises from 14.0% to 35.7% with scale, but InternVL-2 plateaus after 4B and Qwen2.5-VL falls from 21.5% at 32B to 20.3% at 72B.The authors note that differences in language backbones, architecture, and training can outweigh parameter count.Source: §3.9 · pp. 12–13

3.10. The Impact of LLM Backbone Size

  • Backbone size correlates more strongly with conceptual and hypothetical scores (ρ=0.86 and ρ=0.88) than with quantitative scores (ρ=0.64).The authors suggest quantitative reasoning likely requires more accurate visual perception and advanced numerical reasoning, beyond language scaling alone.Source: §3.10 · p. 14

3.11. The Impact of Audio

  • Audio raises overall performance by 2.70% for Gemini-2.5-Pro and 2.80% for Qwen2.5-Omni-7B, a limited improvement relative to the visual contribution.The authors conclude that visual content dominates model performance on this evaluation.Source: §3.11 · p. 14, Table 4 · p. 14

3.12. Error Analysis

  • Error analysis identifies incorrect visual perception (70.68%), inaccurate reasoning progress (63.25%), and lack of domain knowledge (49.40%); most wrong answers involve combined errors.The categories can overlap: the example includes both missed on-screen evidence and a failure to connect potassium sulfate with high humidity.Source: §3.12 · pp. 14–15

3.13. Failure Case Analysis

  • In a failure case, Gemini-2.0-Flash overlooks the video’s “98% relative humidity” label and misreads potassium sulfate’s role, despite chain-of-thought prompting.The correct rationale is that the saturated solution in a sealed desiccator maintains approximately 98% relative humidity during cooling.Source: §3.13 · p. 15

4. Related Work

Prior video-language models extend temporal perception and reasoning, but their scientific reasoning abilities remain underexplored. SciVideoBench builds on video-reasoning benchmarks by focusing on real experimental settings, within a broader motivation to assess AI’s distance from expert-level scientific work.

4.1. Video Large Language Models

  • VideoChatGPT, LLaVA, LongVA, and LongVILA advance video encoding or long-context processing, while scientific reasoning about experiments and measurements remains underexplored.SciVideoBench is introduced to evaluate video reasoning in real-world scientific scenarios.Source: §4.1 · pp. 15, 17

4.2. Video Reasoning Benchmarks

  • SciVideoBench extends video-reasoning benchmarks with real experimental settings and questions about measurement, calculation, and conceptual reasoning.Earlier benchmarks include temporal challenges, long videos, world knowledge, and expert-level instructional or lecture videos.Source: §4.2 · p. 17

4.3. AI for Science

  • Despite advances such as AlphaFold and GNoME, current AI systems remain far from versatile researchers with expert-level depth and accuracy across scientific tasks.SciVideoBench assesses scientific video reasoning requiring visual perception, domain-specific knowledge, and intricate reasoning.Source: §4.3 · p. 17

5. Conclusion

SciVideoBench is presented as a benchmark for PhD-level scientific video reasoning grounded in real-world experimental scenarios. Evaluation finds that advanced models still struggle, while chain-of-thought prompting improves performance.

5. Conclusion

  • SciVideoBench is presented as the first scientific video reasoning benchmark demanding PhD-level knowledge for complex reasoning grounded in real-world experiments.It spans Physics, Chemistry, Biology, and Medicine, with over 20 distinct subjects and quantitative, hypothetical, and conceptual question types.Source: §5 · p. 17
  • Even the most advanced evaluated model, Gemini-2.5-Pro, still struggles with the benchmark’s challenging tasks.The benchmark evaluates 21 models against expert human reasoning.Source: §5 · p. 17
  • Chain-of-thought prompting produces clear performance gains on the benchmark, highlighting the importance of explicit reasoning in complex scientific domains.Source: §5 · p. 18

A. Appendix overview

The appendix provides further details about the approach and additional experimental results, organized across four topics.

A. Appendix overview

B. Disciplines and Subjects

Table 5 lists benchmark subjects grouped by their corresponding disciplines.

B. Disciplines and Subjects

  • Table 5 provides the subject list grouped by discipline.Source: Table 5 · p. 23
  • The table’s organizing principle is the corresponding discipline for each subject.Source: Table 5 · p. 23
  • The table’s scope is the grouping of subjects under their disciplines.Source: Table 5 · p. 23

C. Configuration of Evaluated Models

The evaluated models were released after May 2024, and evaluation follows each model’s official default frame-sampling settings with temperature set to zero.

C. Configuration of Evaluated Models

D. Metadata Generation Using Gemini 2.5 Pro

The section presents Gemini 2.5 Pro prompts for metadata generation and distractor creation, along with an example metadata record.

E. More Failure Case Studies

The failure cases show errors from hallucinated visual features, missed experimental timing, and confusion about components; the conclusion emphasizes grounding reasoning in experimental evidence.

  • InternVL-3-14B hallucinated spectral peaks and voltage-dependent shifts instead of recognizing a single stable peak and confined recombination.Source: E. More Failure Case Studies · p. 29
  • Qwen2.5-VL-32B misread pre-incubation of inhibitors with elastase as a step needed for substrate hydrolysis, although substrate was added later.Source: E. More Failure Case Studies · p. 29
  • InternVL-3-9B misidentified solvents and mishandled stock solutions, overlooking a protocol-grounded total solvent volume of 4.462 mL.Source: E. More Failure Case Studies · p. 29
  • Across the cases, errors reflect hallucinated features, missed temporal sequence, and confusion between experimental component categories; the paper calls for evidence-grounded reasoning.Source: E. More Failure Case Studies · p. 29
Loading 2510.08559v1…