PaperlayerView exact revision on arXiv

Source-linked AI summary

MMR-V: What's Left Unsaid? A Benchmark for Multimodal Deep Reasoning in Videos

Kejian Zhu, Zhuoran Jin, Hongbang Yuan, Jiachun Li, Shangqing Tu, Pengfei Cao, Yubo Chen, Kang Liu, Jun Zhao

arXiv:2506.04141v2cs.CVcs.CL

TL;DR

MMR-V addresses the limited evaluation of long-range multimodal reasoning in videos, where existing benchmarks often rely on question frames and nearby context. It introduces a manually annotated, distractor-controlled benchmark requiring hidden-information inference across video evidence, and finds that current models remain weak, with limited benefit from standard reasoning enhancements. Its coverage is constrained by costly annotation, underrepresented video categories, and predominantly English content.

  • Problem

    Existing video benchmarks mainly emphasize perception and adjacent-frame understanding, leaving long-range multimodal reasoning over hidden information insufficiently evaluated.

  • Method

    MMR-V is a manually annotated benchmark of 317 videos and 1,257 tasks requiring long-range evidence integration, beyond-perception reasoning, and carefully designed distractors.

  • Results

    64.3% accuracy was achieved by the best-performing model, Gemini-2.5-pro, while Chain-of-Thought and scaling test-time compute produced limited gains.

  • Takeaways & Limitations

    MMR-V exposes a substantial gap in current multimodal video reasoning and suggests that effective progress requires deeper analysis across multiple video frames.

  • Takeaways & Limitations

    Scaling is costly because all tasks require manual curation and review, while mystery, puzzle-solving, and gaming categories and non-English languages remain underrepresented.

Abstract

from arXiv · show

The sequential structure of videos poses a challenge to the ability of multimodal large language models (MLLMs) to locate multi-frame evidence and conduct multimodal reasoning. However, existing video benchmarks mainly focus on understanding tasks, which only require models to match frames mentioned in the question (hereafter referred to as "question frame") and perceive a few adjacent frames. To address this gap, we propose MMR-V: A Benchmark for Multimodal Deep Reasoning in Videos. The benchmark is characterized by the following features. (1) Long-range, multi-frame reasoning: Models are required to infer and analyze evidence frames that may be far from the question frame. (2) Beyond perception: Questions cannot be answered through direct perception alone but require reasoning over hidden information. (3) Reliability: All tasks are manually annotated, referencing extensive real-world user understanding to align with common perceptions. (4) Confusability: Carefully designed distractor annotation strategies to reduce model shortcuts. MMR-V consists of 317 videos and 1,257 tasks. Our experiments reveal that current models still struggle with multi-modal reasoning; even the best-performing model, Gemini-2.5-pro, achieves only 64.3% accuracy. Additionally, current reasoning enhancement strategies (Chain-of-Thought and scaling test-time compute) bring limited gains. Error analysis indicates that the CoT demanded for multi-modal reasoning differs from it in textual reasoning, which partly explains the limited performance gains. We hope that MMR-V can inspire further research into enhancing multi-modal reasoning capabilities.

1 INTRODUCTION

MMR-V addresses the lack of long-range multimodal reasoning evaluation in video benchmarks by requiring models to locate and interpret evidence across non-adjacent frames. It combines manually reviewed, confusable tasks with a challenging evaluation of current multimodal models.

  • Motivation: MMR-V targets video reasoning beyond adjacent-frame perception by requiring evidence mining across long-range, non-adjacent frames.Existing benchmarks often use only question frames and nearby context, while MMR-V requires reasoning over hidden implications.
  • Contributions: The benchmark emphasizes long-range multi-frame reasoning, beyond-perception inference, manual reliability, and carefully designed distractors.Tasks are manually annotated and cross-referenced with popular video comments to reduce subjective bias and support realistic judgments.
  • Task framing: MMR-V categorizes tasks into implicit and explicit reasoning based on whether questions require inferring underlying implications beyond surface-level video information.Implicit reasoning interprets subtext, whereas explicit reasoning uses perceivable video information.
  • Benchmark scale: MMR-V contains 317 videos and 1,257 tasks, typically requiring reasoning over about 12 frames spanning roughly 60% of each video.Tasks use multiple-choice questions with approximately ten options on average, and answers are human-annotated and reviewed.
  • Results: 64.3% accuracy was the highest reported result, achieved by Gemini-2.5-pro, indicating substantial difficulty for current multimodal models.The evaluation covered 11 proprietary and 10 open-source models.

2 TASK OVERVIEW

MMR-V defines multimodal reasoning as integrating visual evidence and hidden meanings across a video's temporal span, rather than reasoning only from perceived frames. Its task taxonomy covers implicit and explicit reasoning across six representative task groups.

  • Task motivation: MMR-V requires integrating visual evidence across long-range frames and reasoning about information that is not directly perceivable.This contrasts with text-oriented reasoning based mainly on visual information already perceived from a few frames.
  • Implicit and explicit reasoning: Implicit reasoning interprets hidden meanings beneath surface cues, while explicit reasoning uses multimodal details explicitly presented across long-range frames.Implicit tasks draw on world knowledge with relatively little attention, whereas explicit tasks demand fine-grained perception and effortful reasoning.
  • Implicit reasoning tasks: The implicit task family includes metaphor, theme, emotion, comment, and symbol understanding, plus causal reasoning about underlying relationships.Examples range from interpreting a brown coat metaphor to identifying humor, cultural symbols, author attitude, or why a character acts.
  • Explicit reasoning tasks: The explicit task family includes sequential structure, counterintuitive, cross-modal transfer, and video type-and-intent reasoning.These tasks assess temporal editing, contradictions to common sense, semantic matching beyond the video, and global meta-level interpretation.

3 MMR-V BENCH

MMR-V is constructed as a diverse, manually verified multiple-choice benchmark designed to force multi-frame, deep, realistic reasoning and reduce shortcutting through confusable distractors. Its annotation tests indicate that one strategy produces more confusing tasks.

  • Construction principles: MMR-V construction follows multi-frame, deep-reasoning, and realistic-task principles so answers require dispersed evidence and cannot be obtained through direct perception alone.Video selection avoids straightforward descriptive content and favors creatively edited, thematically rich material.
  • Distractor design: The benchmark uses one correct answer and several carefully crafted distractors generated through three annotation strategies, including model-generated errors and human-written alternatives.The strategies are designed to produce plausible, model-aligned wrong options.
  • Quality evaluation: Strategy 1 produced more confusing distractors and significantly increased task difficulty and quality in a 100-question comparison.When GPT-4o directly answered those tasks, human-verified accuracy was only 17%.
  • Human verification: Five annotators with at least bachelor’s degrees reviewed task accuracy and difficulty using a checklist based on the benchmark’s construction principles.The review process was supported by a dedicated annotation platform and checklist.
  • Dataset statistics: MMR-V comprises 317 videos and 1,257 multiple-choice reasoning tasks spanning six major content domains and fine-grained task categories.Each question has 7–11 candidate answers with exactly one correct option.

4 EXPERIMENTS

MMR-V evaluations show that current models struggle with long-range multimodal reasoning, while adding frames, audio, or reasoning enhancements yields limited or condition-dependent gains.

  • 4.2 MAIN RESULTS: 64.3% was the highest score, achieved by Gemini-2.5-pro with 1 fps sampling; GPT-5 reached 60.9% with fixed input frames.Gemma-3-27b-it was the strongest open-source model but remained behind proprietary systems.
  • 4.2 MAIN RESULTS: CoT prompting added only 0.88% on average, while “Thinking” models improved just 2.4% on MMR-V.Sampled CoTs contained only about 10% visual analysis, indicating predominantly text-based reasoning over questions and options.
  • 4.2 MAIN RESULTS: Larger models outperformed smaller counterparts, with Qwen2.5-VL-72B at 39.1% versus 30.1% for 7B, and GPT-4o at 52.8% versus 34.8% for GPT-4o-mini.The corresponding relative gains were 9% and 18%.
  • 4.2 MAIN RESULTS: Models scored 7.9% higher on implicit than explicit tasks, because implicit videos provide distributed cues whereas explicit tasks require locating a few specific evidence frames.The explicit lighter example illustrates the finer-grained localization burden.
  • 4.2 MAIN RESULTS: Counterintuitive Reasoning, Sequential Structure Reasoning, and Comment Matching were especially difficult, reflecting weak multi-frame evidence localization and limited capture of humor or emotion.CIR and SSR require long-range reasoning rather than reliance on the question frame, while CM exposes a model-human gap in implicit reasoning.
  • 4.3 INFLUENCE OF FRAMES COUNT: Accuracy increased with more input frames but gradually plateaued, with early gains from added evidence frames and later limits from weak multi-frame reasoning.Implicit-task performance continued improving because relevant cues were dispersed, whereas explicit clues were fewer and more localized.
  • 4.4 INFLUENCE OF AUDIO INPUT: Adding audio improved accuracy by 1.4% for Gemini 2.0-Flash and 1.0% for both Gemini 2.0-Flash-Thinking and Phi-4-multimodal-instruct.The comparison evaluates whether audio contributes to reasoning for models supporting full-modal input.
  • 4.5 ERROR ANALYSIS: GPT-4o errors included missing visual reasoning, implicit misinterpretation, knowledge insufficiency, and reasoning errors.The visual-reasoning category involved failure to locate correct evidence frames and perform long-range, multi-frame analysis.

5 RELATED WORK

Existing video benchmarks emphasize perception and understanding, while multimodal reasoning evaluations remain largely text- or image-oriented. Recent video reasoning benchmarks broaden coverage, but target automated CoT generation, specialized knowledge, or narrative reasoning.

  • Existing video benchmarks primarily evaluate perception and understanding tasks such as action recognition and video description.
  • Multimodal reasoning evaluations still focus mainly on text-based reasoning, with limited assessment of MLLMs.
  • Recent video reasoning benchmarks target automated CoT generation, specialized knowledge reasoning, or narrative reasoning in long videos.

6 CONCLUSION

MMR-V is a human-annotated benchmark for multimodal deep reasoning in videos that exposes a substantial gap between current models and humans. Higher-performing models analyze videos more extensively, motivating multi-frame reasoning and tool use as promising directions.

  • The best model still trails human accuracy by 21.7%, revealing a substantial gap in interpreting and reasoning about video information.
  • Models with higher MMR-V accuracy tend to perform more extensive and in-depth video analysis.
  • MMR-V is proposed as a reliable evaluation benchmark for developing MLLMs and advancing multimodal reasoning research.

ETHICS STATEMENT

The authors report that human-participant procedures followed ethical guidelines, with informed consent, compensation, safety and bias review, and planned public release under CC-BY 4.0.

  • Human-participant procedures followed relevant ethical guidelines, and compensated internal annotators provided informed consent.
  • The dataset was reviewed for safety and potential biases and prepared for public release under the CC-BY 4.0 license.

REPRODUCIBILITY STATEMENT

MMR-V documents its construction workflow, annotation checklist, supporting figures, and reproducibility materials. The checklist requires full-video viewing, long-distance reasoning, and alignment with public interpretations, while annotation uses GPT-4o-assisted distractor generation.

  • The supplementary materials provide experiment code and setup details, while Figures 7 and 8 document the construction pipeline and annotation platform.
  • Scaling the benchmark remains costly because tasks and answers require human curation and review, and multilingual applicability is constrained by predominantly English videos.
  • C.1 CHECKLIST: Annotators must watch each entire video, require long-distance multi-frame reasoning, and consult official interpretations or user consensus to reduce bias.
  • The construction process is organized into video collection, data annotation, and quality assurance stages.
  • GPT-4o assists annotation by generating interference options, and the benchmark distinguishes implicit from explicit reasoning tasks.

C.3 PIPELINE OF ERROR ANALYSIS

The benchmark’s error-analysis pipeline uses representative anchors and external review to standardize failure categories and reduce author bias. It also documents broad video diversity and compares MMR-V with existing benchmarks.

  • C.3 PIPELINE OF ERROR ANALYSIS: Representative anchor examples define six failure categories and help evaluators distinguish boundary cases during error analysis.Non-author evaluators review the definitions to mitigate author bias.
  • C.3 PIPELINE OF ERROR ANALYSIS: MMR-V spans six video categories, including Life, Animation, Film, Art, TV, and Philosophy, with subcategories for finer classification.Video lengths range from 7 to 3771 seconds.
  • C.3 PIPELINE OF ERROR ANALYSIS: Table 5 compares MMR-V with existing video benchmarks using average duration and annotation method.Annotation methods are labeled automatic, manual, or combined.

E TASK DETAILS

MMR-V organizes reasoning tasks into hierarchical levels spanning implicit and explicit reasoning, ten task classes, and 33 subclasses. Examples emphasize metaphor, philosophy, reflection, emotion, and overall-content interpretation.

  • TASK TAXONOMY: The task taxonomy has three levels: implicit versus explicit reasoning, ten task classes, and 33 task subclasses.The hierarchy captures different reasoning types and finer-grained task distinctions.
  • I. METAPHOR UNDERSTANDING (MU): Metaphor Understanding tasks require interpreting relationships between visual entities or environments, including structural, orientational, creative, and broader conceptual metaphors.Examples ask what a brown coat symbolizes, why a dance ends with descent, and what a pink fairy ball represents.
  • DATA PROCESSING: Original source videos are processed to remove frames that contain shortcuts to the answers.The linked videos are source versions, whereas benchmark videos are trimmed for evaluation.
  • II. HIGH-LEVEL CONCEPTS: Philosophical and reflective tasks ask models to infer a video’s overall message, social critique, or personally resonant concept.Examples include stress relief, social satire, and avoiding premature judgment.
  • II. HIGH-LEVEL CONCEPTS: Some tasks require interpreting implied meaning across scenes rather than relying on a single visible event.One example asks what the contrast between scenes around 0:47 and 1:11 implies.
  • III. OTHER TASKS: Task examples also test holistic content control, such as proposing a suitable title and analyzing characters’ emotions or artistic cues.The benchmark includes title selection and emotion-related questions involving facial expression, body movement, lighting, and narrative meaning.

F EVALUATION DETAILS

The evaluation covers a broad set of proprietary and open-source baseline models under standardized multi-image settings. Frame counts are fixed per model and sampled evenly throughout each video.

  • F EVALUATION DETAILS: The baselines include GPT, Gemini, Claude, o-series, Qwen, and Gemma model families across proprietary and open-source systems.The listed models include GPT-4o, GPT-4.1, GPT-5, Gemini-2.5, o3, o4-mini, and Qwen2.5-VL variants.
  • F EVALUATION DETAILS: Models supporting multi-image input follow official or prior-work configurations, with a fixed frame count for each model.Frames are sampled evenly across the video duration.

G FURTHER RESULTS

Further analyses show that performance generally declines with video length, while long-context models are more stable. Audio provides complementary cues that can resolve visual ambiguity, recover wordplay, and convey mood or narration.

  • G.1 IMPACT OF VIDEO LENGTH ON MODEL PERFORMANCE: Performance generally decreases as video length increases, but long-context models show a smaller gap across short, medium, and long videos.The duration groups contain 329, 593, and 335 tasks; Gemini-2.0-Flash and Gemini-2.5-Pro remain comparatively stable.
  • G.2 BENEFIT OF AUDIO INPUT: Audio can correct a misleading visual interpretation by revealing that a seemingly quiet library is noisy until the protagonist puts on headphones.The example contrasts visual-only inference with audio-grounded interpretation.
  • G.2 BENEFIT OF AUDIO INPUT: Audio can disambiguate homophones, such as the relationship between “parachute” and “pair of shoes” in a video example.This demonstrates a reasoning cue unavailable from the visual stream alone.
  • G.2 BENEFIT OF AUDIO INPUT: Background music can communicate overall mood, while voice narration supplies additional information in magic-performance videos.Audio therefore contributes cues from perspectives beyond the visible scene.

H DETAILS OF COT ANALYSIS EXPERIMENT

The CoT analysis experiment separates reasoning into text, question-frame, and other-frame video analysis, comparing zero-shot and zero-shot-plus-CoT evaluation. Case studies contrast deep video-grounded reasoning with shallow video perception and textual reasoning.

  • CoT Analysis Setup: The analysis categorizes reasoning into text analysis, question-frame analysis, other-frame analysis, and broader video analysis.Question-frame analysis targets the frame referenced in the question, while other-frame analysis covers frames outside it.
  • Evaluation Settings: The experiment evaluates models in both zero-shot and zero-shot-plus-CoT settings using dedicated evaluation prompts.The appendix provides separate prompts for the zero-shot and zero-shot-plus-CoT conditions.
  • Case Studies: The case-study section uses representative reasoning processes and error cases to expose shortcomings in current multimodal reasoning.The examples include lack of visual reasoning, implicit misinterpretation, reasoning errors, and a correct case.
  • CoT Comparison: Deep multimodal reasoning grounded in video content reaches the correct answer, whereas shallow video perception combined with mainly textual reasoning leads to an incorrect outcome.The comparison distinguishes the upper CoTs, which analyze video deeply, from the bottom CoT, which relies primarily on text and limited video perception.
Loading 2506.04141v2…