Source-linked AI summary
MMOU: A Massive Multi-Task Omni Understanding and Reasoning Benchmark for Long and Complex Real-World Videos
Arushi Goel, Sreyan Ghosh, Vatsal Agarwal, Nishit Anand, Kaousheik Jayakumar, Lasha Koroshinadze, Yao Xu, Katie Lyons, James Case, Karan Sapra, Kevin J. Shih, Siddharth Gururani, Abhinav Shrivastava, Ramani Duraiswami, Dinesh Manocha, Andrew Tao, Bryan Catanzaro, Mohammad Shoeybi, Wei Ping
TL;DR
Existing evaluations provide limited evidence about joint audio-visual reasoning in long, complex videos. MMOU addresses this gap with a large, multi-skill benchmark and evaluates more than 20 multimodal models, finding substantial performance gaps and frequent failures on fundamental long-video reasoning tasks.
Problem
Existing benchmarks and models largely emphasize isolated modalities or short videos, leaving joint audio-visual reasoning over long, complex real-world videos insufficiently evaluated.
Method
MMOU evaluates 20,000 professionally annotated questions from 11,877 long-form videos across 13 skills requiring integrated audio-visual reasoning over time.
Results
The strongest closed-source model achieves 64.2% accuracy while the strongest open-source model reaches 46.8%, and modality-restricted or cascaded systems perform inadequately.
Takeaways & Limitations
Current multimodal models still struggle with basic audio-visual reasoning in long real-world videos, revealing a substantial gap from human performance.
Takeaways & Limitations
Because MMOU uses publicly available web videos and multiple-choice evaluation, it may contain train-test leakage or content biases and does not fully capture open-ended reasoning.
Abstract
from arXiv · showhide
Multimodal Large Language Models (MLLMs) have shown strong performance in visual and audio understanding when evaluated in isolation. However, their ability to jointly reason over omni-modal (visual, audio, and textual) signals in long and complex videos remains largely unexplored. We introduce MMOU, a new benchmark designed to systematically evaluate multimodal understanding and reasoning under these challenging, real-world conditions. MMOU consists of 20,000 carefully curated questions paired with 11877 web-collected videos of varying length, spanning diverse domains and exhibiting rich, tightly coupled audio-visual content. The benchmark covers 13 fundamental skill categories, all of which require integrating evidence across modalities and time. All questions are manually annotated across multiple turns by professional annotators, ensuring high quality and reasoning fidelity. We evaluate 20+ state-of-the-art open-source and proprietary multimodal models on MMOU. The results expose substantial performance gaps: the best closed-source model achieves only 64.2% accuracy, while the strongest open-source model reaches just 46.8%. Our results highlight the challenges of long-form omni-modal understanding, revealing that current models frequently fail to apply even fundamental skills in long videos. Through detailed analysis, we further identify systematic failure modes and provide insights into where and why current models break.
1 Introduction
Existing multimodal models perform strongly on isolated audio and visual tasks but often fail to integrate both modalities. MMOU addresses this gap with a large benchmark for joint audio-visual reasoning in long, complex real-world videos.
- Existing MLLMs often fail to jointly perceive and reason across audio and visual modalities.The imbalance of single-modality training data and benchmarks limits unified multimodal integration.
- Current video evaluations rarely test tightly coupled audio-visual reasoning across extended time horizons.Many benchmarks ignore audio, treat it as auxiliary, or focus on short clips.
- MMOU evaluates joint audio-visual understanding across 13 fundamental skills in long and complex real-world videos.Each question requires simultaneous integration of audio and visual information, so removing either modality leads to failure.
- MMOU contains 20,000 expertly annotated multiple-choice questions sourced from 11,877 long-form videos spanning 10 domains and 35 fine-grained subcategories.The benchmark uses hard distractors and professionally trained human annotation.
2 Related Work
Related benchmarks have advanced isolated audio, visual, and video evaluation, but they incompletely capture omni-modal reasoning. In particular, joint reasoning across modalities, tasks, and temporal context remains underrepresented.
- Existing multimodal evaluation protocols remain largely unimodal or narrowly task-defined.Such settings do not capture the complexity of real-world multimodal reasoning.
- Strong results on individual benchmarks do not necessarily translate to robust omni-modal understanding.Robust understanding requires joint reasoning across modalities, tasks, and temporal context.
- Many video benchmarks ignore audio or treat it as auxiliary while emphasizing short clips.This limits evaluation of long-term temporal dependencies and synchronized audio-visual understanding.
3 MMOU
MMOU is a large, diverse benchmark of long-form videos and multi-skill questions designed to require synchronized audio-visual reasoning over time. Its construction combines broad coverage, temporal diversity, expert curation, and hard distractors.
- Dataset Statistics: MMOU contains 20,000 multiple-choice QA pairs from 11,877 web-sourced long-form videos.Test and test-mini videos average 522.9 and 754.8 seconds, respectively, with durations extending to thousands of seconds.
- Dataset Statistics: The videos cover 10 major categories and 35 fine-grained subcategories, while questions span 13 skill types averaging three skills per question.Questions are converted from open-ended annotations into ten-option multiple-choice items with nine hard distractors.
- Dataset Statistics: MMOU distributes answer-relevant evidence across diverse temporal locations to reduce positional bias.The average answer position is 302.36 seconds, and evidence is distributed relative to video length.
- Benchmark Comparison: MMOU is compared with prior benchmarks as a long-form synchronized audio-visual video benchmark spanning durations from seconds to several hours.Prior omni-modal benchmarks generally cover substantially shorter and less complex videos.
- Data Collection, Curation & Annotation: A structured expert-driven pipeline filters questions for ambiguity, redundancy, triviality, timestamp misalignment, and weak audio-visual grounding.Questions are authored openly, converted using nine hard distractors, and quality-checked before inclusion.
- Data Collection, Curation & Annotation: The benchmark requires reasoning about temporal dependencies, sub-scenes, holistic video themes, inference, localization, grounding, counting, interactions, and edited-segment continuity.These skills are explicitly designed to integrate audio and visual information across long videos.
4 Experimental Setup
MMOU is evaluated with diverse omni-modal, modality-restricted, text-only, and cascaded baselines using micro-averaged accuracy. The setup tests whether models require joint audio-visual inputs rather than captions or textual biases alone.
- Baselines: The evaluation spans omni-modal, audio-only, vision-only, and text-only models.This design enables comparison across models with different modality access.
- Baselines: Closed-source baselines include Gemini 2.5 Flash and Gemini 2.5 Pro, alongside open-source omni-modal models.The benchmark includes models explicitly designed to jointly process audio and visual inputs.
- Baselines: Modality-restricted baselines isolate the contributions of visual and audio cues.Vision-only and audio-only models are evaluated separately on MMOU.
- Baselines: Text-only and cascaded caption-based baselines test whether language biases or separately generated captions can replace direct multimodal perception.Cascaded systems provide separately generated audio and visual captions to the language model.
- Evaluation: Models are scored with micro-averaged accuracy after selecting one answer option, with randomized option order and multiple prompt variants.The reported configuration uses majority selection across five option-order randomizations and the best prompt configuration.
5 Results and Discussion
MMOU evaluations show substantial gaps in multimodal reasoning: closed-source models lead, but performance remains below human level, and both modalities are necessary for strong results.
- 64.2% overall accuracy makes Gemini 2.5 Pro the strongest baseline across diverse video domains and durations.Open-source audio-visual models show a relative performance drop exceeding 24.7% compared with Gemini 2.5 Pro.
- 44% accuracy from video-only Qwen3-VL-32B demonstrates that visual input alone is insufficient for MMOU.Audio-only settings also produce significant performance drops for Audio Flamingo 3 and Qwen3-Omni-30B-A3B.
- Text-only and cascaded caption-based models do not achieve state-of-the-art performance, supporting the need for joint end-to-end audio-visual reasoning.
6 Results Analysis
Analysis reveals weaknesses in temporal and basic skills, sensitivity to late evidence, and a persistent gap between multiple-choice recognition and open-ended grounded answering.
- Skill-wise Performance Analysis: Closed models outperform open models across most skills, yet all models remain weak on temporal understanding and counting.The same weakness appears on the test-mini subset.
- Temporal Position Sensitivity: Accuracy steadily declines when relevant evidence appears later in a video, with a sharp drop near the end of long sequences.The pattern holds across both open and closed models.
- Open-Ended Evaluation: Gemini 2.5 Pro leads open-ended evaluation with a weighted overall score of 3.90, ahead of Qwen3-Omni-30B-Instruct at 2.86.
- Open-Ended Evaluation: Gemini 2.5 Pro and Qwen3-Omni-30B show stronger Faithfulness and Clarity than Correctness and Completeness in open-ended responses.For Gemini 2.5 Pro, Correctness is 3.71 versus Faithfulness 3.80 and Clarity 4.62; Completeness is 3.86.
- Open-Ended vs Multiple-Choice: Models answer more questions correctly in multiple-choice format than in low-correctness open-ended cases, indicating recognition benefits from answer options.
7 Conclusion, Limitations and Future Work
MMOU introduces a broad benchmark for long-video omni-modal reasoning and finds that current models struggle with basic audio-visual reasoning. The benchmark is limited by web-video bias, possible leakage, and reliance on multiple-choice evaluation.
- Conclusion: MMOU evaluates omni-modal understanding and reasoning across long, complex real-world audio-visual videos.The benchmark is designed around joint audio-visual perception and diverse reasoning skills.
- Conclusion: Extensive evaluations show a substantial gap between current multimodal models and human performance on basic audio-visual reasoning over long videos.
- Limitations: Web-collected videos may introduce content bias and potential train–test leakage for closed and open-weight models.
- Limitations and Future Work: Multiple-choice evaluation does not fully capture real-world open-ended reasoning.Future work includes stronger open-ended protocols and broader coverage of unstructured real-world videos.
- Benchmark Construction: The benchmark uses expert annotators, timestamped answer evidence, multimodal questions, task labels, and audits of 20% of question–answer pairs.Questions are required to assess video and audio simultaneously, with recommended question counts varying by video length.
D Human Evaluation on MMOU
MMOU evaluates models across modalities using both multiple-choice and open-ended protocols, with open-ended answers scored by a four-criterion LLM-judge rubric. The evaluation includes proprietary, open-source, unimodal, cascaded, and text-only baselines under identical question sets and protocols.
- Model Evaluation: MMOU compares proprietary, open-source, vision-only, audio-only, cascaded, and text-only models under identical question sets and evaluation protocols.The evaluated systems include Gemini models, omni-modal open-source models, unimodal baselines, cascaded caption-fusion approaches, and text-only LLMs.
- Open-Ended Evaluation: Open-ended evaluation requires free-form answers without predefined options, testing whether models can articulate knowledge beyond recognition or option elimination.The protocol is intended to distinguish underlying knowledge from multiple-choice recognition and elimination strategies.
- Evaluation Rubric: Each open-ended response receives 1–5 scores for correctness, completeness, faithfulness, and clarity.Correctness measures alignment with the reference answer; completeness measures coverage; faithfulness measures unsupported claims; and clarity measures directness and understandability.
- Evaluation Rubric: The weighted overall score emphasizes correctness while assigning equal weight to completeness, faithfulness, and clarity collectively.The passages state that correctness receives half the weight and the remaining three dimensions share the other half; scores lie in [1, 5].
- LLM Judges: GPT-5 and a custom Qwen-3.5-0.8B judge receive the question, reference answer, audio-visual caption, and model response for structured scoring.The custom judge uses the same four criteria and score anchors as GPT-5, outputting a score and brief reason per dimension in JSON.
H.2 Distractor Quality Analysis
MMOU’s distractor analysis shows that model errors are usually caused by plausible options grounded in the video rather than fabricated alternatives. This indicates that rejecting incorrect but relevant audio-visual content remains difficult.
- Distractor Types: 77–91% of wrong answers select in-video distractors, compared with 44.4% expected under random selection.There are four in-video and five out-of-video distractors among nine distractors per question.
- Distractor Types: In-video distractors reference real audio-visual events and are designed to confuse models that partially understood the video.Out-of-video distractors are plausible but fabricated content absent from the video.
- Interpretation: Only 9–23% of wrong answers select out-of-video distractors, suggesting fabricated alternatives are more often filtered out than wrong-but-grounded options.The analysis attributes the dominant error pattern to the need for precise audio-visual understanding to reject in-video distractors.
I Compute Resources
MMOU evaluation uses high-end GPU servers for local open-source models and APIs for closed-source and selected backbone models. Full-test evaluation of larger models requires substantial but bounded compute.
- Hardware and Access: Evaluations use NVIDIA A6000 and A100 GPUs, with open-source models run locally on eight GPUs per model.Gemini, text-only, and cascaded backbones are accessed through their respective APIs.
- Compute Cost: For models with 30B+ parameters, evaluating the 15,000-question full test set takes approximately 5–6 hours, or 44 GPU-hours per model.The reported runtime applies to the full test set and larger models.
K Dataset Quality Control and Safeguards
MMOU applies multi-stage quality control and content-safety screening to its web-collected videos and annotated questions. Reviewers audit question relevance, language, task labels, and answer correctness before release.
- Quality Control: Reviewers audit 20% of question–answer pairs for video relevance, grammar, skill-type accuracy, and answer correctness.Questions failing any criterion are revised or discarded, with feedback iterated between reviewers and annotators.
- Content Safety: All collected videos undergo manual review before question generation, and harmful, offensive, or sensitive content is excluded.The videos come from publicly available platforms and are used for non-commercial research under their Terms of Service.