PaperlayerView exact revision on arXiv

Source-linked AI summary

OmniVideoBench: Towards Audio-Visual Understanding Evaluation for Omni MLLMs

Caorui Li, Yu Chen, Yiyan Ji, Jin Xu, Zhenyu Cui, Shihao Li, Yuanxing Zhang, Wentao Wang, Zhenghao Song, Dingling Zhang, Ying He, Haoxiang Liu, Yuxuan Wang, Qiufeng Wang, Jiafu Tang, Zhenhe Wu, Jiehui Luo, Zhiyu Pan, Weihao Xie, Chenchen Zhang, Zhaohui Wang, Jiayi Tian, Yanghai Wang, Zhe Cao, Minxin Dai, Ke Wang, Runzhe Wen, Yinghao Ma, Yaning Pan, Sungkyun Chang, Termeh Taheri, Haiwen Xia, Christos Plachouras, Emmanouil Benetos, Yizhi Li, Ge Zhang, Jian Yang, Tianhao Peng, Zili Wang, Minghao Liu, Junran Peng, Zhaoxiang Zhang, Jiaheng Liu

arXiv:2510.10689v3cs.AI

TL;DR

Existing benchmarks inadequately evaluate audio-visual reasoning across long videos and complementary modalities. OmniVideoBench evaluates this capability with reasoning-annotated video questions, and the best-performing model scores 58.90%, below the benchmark’s 60% passing threshold.

  • Problem

    Existing audio-visual reasoning evaluation remains underdeveloped, with benchmarks often using short clips that underrepresent long-term temporal dependencies or emphasizing a single modality.

    Source: §1 · p. 1

  • Method

    OmniVideoBench evaluates audio-visual collaborative reasoning using 1,000 reasoning-annotated question-answer pairs derived from 628 diverse videos, with answers supported by explicit reasoning steps.

    Source: §1 · p. 1, §2.1 · p. 2

  • Results

    58.90% accuracy is the best score on OmniVideoBench, below its 60% passing threshold; open-source models perform close to random except Qwen3-Omni.

    Source: §1 · p. 1

  • Takeaways & Limitations

    Both open- and closed-source models still struggle with modality complementarity, long-form temporal reasoning, and music understanding.

    Source: §5 · p. 10

Abstract

from arXiv · show

Recent advances in multimodal large language models (MLLMs) have demonstrated substantial potential in video understanding. However, existing benchmarks fail to comprehensively evaluate synergistic reasoning capabilities across audio and visual modalities, often neglecting either one of the modalities or integrating them in a logically inconsistent manner. To bridge this gap, we introduce OmniVideoBench, a large-scale and rigorously designed benchmark dedicated to assessing synergistic audio-visual understanding, with a strong emphasis on modality complementarity and logical consistency. Specifically, OmniVideoBench comprises 1000 high-quality question-answer(QA) pairs, each annotated with step-by-step reasoning traces, derived from 628 diverse videos ranging from several seconds to 30 minutes, and manually verified to guarantee complete correctness and uniqueness. Moreover, OmniVideoBench encompasses 13 carefully designed question types, covering temporal reasoning, spatial localization, counting, causal inference, summarization, and beyond, thereby capturing the essential challenges of video understanding. Evaluation of multiple MLLMs on OmniVideoBench reveals a pronounced gap between model performance and human reasoning, with open-source models lagging significantly behind their closed-source counterparts, underscoring the inherent difficulty of genuine audio-visual reasoning. We will release OmniVideoBench to foster the development of MLLMs with stronger and more generalizable reasoning capabilities.

1 Introduction

Prior benchmarks often focus on short clips or emphasize a single modality (Li et al., 2024a; Hong et al., 2025), missing the challenge of integrating long temporal sequences and complementary audio-visual evidence consistently. OmniVideoBench addresses this gap with a large-scale benchmark designed to evaluate audio-visual reasoning in MLLMs.

Benchmark construction

  • OmniVideoBench contains 1,000 QA pairs from 628 diverse videos spanning up to 30 minutes, each annotated with step-by-step reasoning chains that identify modality and evidence.The videos cover 8 major categories and 68 subcategories, including news, sports, documentaries, vlogs, and ego-centric recordings.Source: §1 · p. 1

Evaluation findings

  • 58.90% accuracy is the best score on OmniVideoBench, achieved by Gemini-2.0-Pro; no model passes 60%, and open-source models are near random except Qwen3-Omni.The results indicate that current MLLMs still find audio-visual reasoning challenging.Source: §1 · p. 1
  • Long-video performance remains uneven: Gemini-2.5-Pro is relatively robust, while Gemini-2.0-Flash and Qwen3-Omni-30B-A3B still struggle.The findings point to substantial room for improvement in omni-understanding of long videos.Source: §1 · p. 2
  • Gemini-2.5-Pro accuracy varies by audio signal, reaching 38.46% on music, 57.72% on sound, and 61.66% on speech videos.Source: §1 · p. 2
  • Gemini-2.5-Pro scores below 50% on background and music understanding but above 80% on relationship reasoning and summarization.Background and music understanding requires attention to low-semantic acoustic cues such as musical style and tempo changes.Source: §1 · p. 2

2 OmniVideoBench

OmniVideoBench evaluates whether MLLMs can combine video, audio, and text to produce answers supported by explicit reasoning steps. Its design targets complementary audio-visual reasoning across objects, scene dynamics, and context.

2.2 Video Collection

  • Videos from YouTube and Bilibili span eight broad categories, nearly 70 subcategories, and durations from several seconds to 30 minutes.The collection emphasizes quality and rich audio-visual content, controls the distribution of categories with dense audio, and restricts selection to recent publications to avoid overlap with training data.Source: §2.2 · p. 3

2.3 Data Annotation

  • The benchmark uses 13 question types, with questions requiring audio-visual reasoning and correct, unique answers rather than resolution- or frame-rate-dependent cues.Design rules reduce textual cues, cap answer length, balance option formats, use relevant distractors, and keep semantic distances between options consistent.Source: §2.3 · pp. 3–4

2.4 Quality Assurance

  • After unimodal and text-only filtering and final review, the authors retained 1000 QA pairs with atomic, step-by-step reasoning chains.Filtering reduced approximately 2,500 initial pairs to 1,500 after unimodal filtering and 1,103 after text-only filtering; each reasoning step specifies modality, evidence, and inference.Source: §2.4 · p. 4

2.5 Dataset Statistics

  • The dataset contains 628 real-world videos and 1000 QA pairs across 13 task types, with reasoning chains averaging 5.68 steps and 54% of steps grounded in vision.The remaining 46% of reasoning steps are grounded in audio; 762, 147, and 91 QA pairs relate to Speech, Sound, and Music, respectively.Source: §2.5 · p. 5

2.6 Dataset Comparison

  • With audio disabled, Gemini-2.0-Flash falls to near-random performance, while Qwen2.5-Omni-7B performs closer to random guessing on OmniVideoBench.The benchmark targets diverse video types, broader temporal spans, and fine-grained cross-modal reasoning.Source: Figure 4 · p. 5, §2.6 · p. 6

3 Experiments

Experiments show that current MLLMs trail human performance and reveal persistent weaknesses in music-related and long-video reasoning. Further analyses probe task-level performance, audio integration, frame sampling, and question format.

3.1 Baseline Models

  • The evaluation compares open-source and closed-source omni MLLMs, alongside Qwen2.5-VL models and DeepSeek-V3.1.The open-source set includes Qwen Omni, Baichuan-Omni-1.5, HumanOmni, MiniCPM-o, VideoLLaMA2, VITA-1.5-7B, and OmniVinci-9B.Source: §3.1 · p. 6

3.4 Further Analysis

  • Gemini-2.5-Pro achieves the highest accuracy on 11 of 13 tasks, yet Background and Music Understanding remains hardest, below 50% even for it.Relationship Reasoning and Summarization are comparatively easier; the hardest category requires connecting low-semantic acoustic cues with higher-level reasoning.Source: Figure 5 · p. 7, §3.4 · p. 8

3.2 Human Performance

  • Human annotators achieved 82.69% accuracy, establishing a level that existing models still fall significantly short of.Ten annotators’ answers to 50 shared questions differed by no more than five, supporting the viability of the manual evaluation.Source: §3.2 · p. 7

3.3 Main Results

  • Open-source MLLMs lag closed-source models, Gemini-2.5-Pro leads on most tasks, and most models struggle with music-dominated and long videos.Music-dominated videos are harder than those with human voices or ambient sounds; Gemini-2.5-Pro is relatively robust on long videos, while most MLLMs struggle.Source: §3.3 · pp. 7–8

3.4 Further Analysis

  • Open-ended QA significantly lowers every evaluated model’s accuracy versus MCQ; Gemini-2.5-Pro declines by over 14% relatively, and open-source models drop more.Open-ended questions remove answer options, preventing random guessing and eliminating lexical cues from candidate choices.Source: Table 4 · p. 8, §3.4 · p. 8
  • ASR transcripts improve all tested models over visual-only inputs, but Qwen2.5-Omni-7B underperforms parameter-matched Qwen2.5-VL-7B, and ASR poorly supports music or sound reasoning.The results indicate that speech recognition can help on some tasks, but cannot replace deeper audio comprehension.Source: §3.4 · p. 8, Figure 6 · p. 9
  • Increasing frame counts from 32 to 256 steadily improves Qwen2.5-Omni-7B and Qwen3-Omni-30B-A3B accuracy, with larger gains on longer videos.The authors attribute the gains to richer temporal coverage, which captures motion cues and supports cross-modal alignment.Source: §3.4 · p. 8, Figure 7 · p. 9

4 Related Works

Related work traces MLLMs’ shift from vision-language integration toward omni-modal systems and reviews benchmarks progressing from perception to cognitive, domain-specific, and long-context video evaluation.

Omni-Understanding MLLMs

  • MLLMs evolved from vision-language integration toward Any-to-Any systems that unify modalities in an LLM semantic space and increasingly use natively multimodal architectures.GPT-4o exemplifies the native multimodal approach (Hurst et al., 2024).Source: §4 · p. 9
  • State-of-the-art omni-modal models pioneer end-to-end streaming across video and audio, while Gemini models demonstrate strong performance with native multimodality and massive context windows.These capabilities support simultaneous video and audio processing to generate text and speech (Xu et al., 2025c; Zhao et al., 2025; Li et al., 2024c; 2025; Yao et al., 2024; Sun et al., 2025; Liu et al., 2025b; Wu et al., 2025a).Source: §4 · p. 9

MLLM Benchmarks

  • MLLM evaluation has expanded from perception benchmarks to cognitive reasoning frameworks and specialized finance and medicine assessments.Examples include MLLM-Bench’s hierarchy of cognitive skills and the expert-level multidisciplinary reasoning challenges in MMMU and MMMU-Pro (Ge et al., 2025; Yue et al., 2023; 2024b).Source: §4 · p. 9
  • Video benchmarks increasingly test long-context temporal understanding, exposing limitations in current models.The review describes this as a critical challenge for video evaluation (Liu et al., 2025c).Source: §4 · p. 9

5 Conclusion

OmniVideoBench evaluates audio-visual collaborative reasoning in MLLMs with diverse videos, verified QA pairs, and explicit reasoning annotations. Experiments find persistent model weaknesses and a large gap from human-level performance.

5 Conclusion

  • OmniVideoBench is a large-scale benchmark for evaluating audio-visual collaborative reasoning in MLLMs.It combines diverse videos, carefully verified QA pairs, and explicit reasoning annotations.Source: §5 · p. 10
  • Experiments show that open- and closed-source models struggle with modality complementarity, long-form temporal reasoning, and music understanding.The results underscore a large gap from human-level performance.Source: §5 · p. 10
  • The authors hope OmniVideoBench will drive research toward more robust and generalizable multimodal reasoning systems.Source: §5 · p. 10

6 Contributions

The contributions section identifies data annotation management and data quality inspection as team responsibilities, and lists the core contributors and their affiliations.

6 Contributions

  • The team identifies data annotation management as a contribution area.The passage names this responsibility but provides no further description of its work.Source: §6 · p. 11
  • The team identifies data quality inspection as a contribution area.The passage names this responsibility but provides no further description of its work.Source: §6 · p. 11

Core Contributors

  • The core contributor roster spans six listed organizations and institutions.It names Jin Xu, Zhenyu Cui, Shihao Li, Yuanxing Zhang, Wentao Wang, Zhenghao Song, Dingling Zhang, Ying He, Haoxiang Liu, Yuxuan Wang, and Qiufeng Wang, with affiliations listed in the passage.Source: Core Contributors · p. 11

A Full Video Category Taxonomy

OmniVideoBench’s videos span 8 major categories and 68 subcategories.

A Full Video Category Taxonomy

B Detailed Principles of Video Collection

Video selection aimed to support objective evaluation through diverse, dynamic content and complementary audio-visual information, avoiding cases where either modality made the other redundant. Collection rules also addressed training-data overlap and excluded content that could undermine genuine multimodal reasoning.

Video publication date

  • Videos were restricted to publication after June 2024 to reduce training-data overlap, answer memorization, and potential overestimation.The benchmark selected the most recent videos possible because existing training datasets may contain YouTube videos or overlapping content such as TV shows.Source: B Detailed Principles of Video… · p. 17

Rich dynamic visual information

  • Videos needed sufficient dynamic visual content; largely static scenes or minimal motion confined to a small frame area were excluded.The visual stream had to contain dynamic information that models could capture and analyze.Source: B Detailed Principles of Video… · p. 17

Effective audio information

  • Audio had to align with the visual content, whether it consisted of speech, environmental sound, or music; unrelated background tracks were invalid.This requirement was intended to support fair evaluation of audio-visual collaborative reasoning.Source: B Detailed Principles of Video… · p. 17

Absence of subtitle

  • Videos with embedded subtitles or large text overlays were excluded because they could reveal audio information or events through vision alone.Such text could disclose characters’ speech, mental states, or ongoing events, undermining assessment of genuine understanding and reasoning.Source: B Detailed Principles of Video… · pp. 17–18

Video resolution

C Prompts Used in This Work

Appendix C provides prompts for direct multiple-choice evaluation, filtering questions for option dependence, and eliciting step-by-step reasoning before a final choice.

C.1 Prompt for Overall Evaluation

  • The overall-evaluation prompt asks the model to answer from the video and return only the selected option’s letter.Source: §C.1 · p. 18
  • If video access is limited, the prompt still requires choosing the option considered most likely, without giving another reason for not choosing.Source: §C.1 · p. 18

C.2 Prompt to select questions that can be answered without relying on options

  • The option-dependence prompt assigns an impartial judge to decide whether a question inherently requires its multiple-choice options, not to answer the question.The video is omitted, but the judge is told to assume full knowledge of it.Source: §C.2 · p. 18
  • The judge returns “No” when a question remains reasonably answerable without options, even if its wording might change slightly.Source: §C.2 · p. 18
  • The judge returns “Yes” when a question cannot be answered without its options, such as one asking “Which of the following...”.Source: §C.2 · p. 18
  • The option-dependence prompt takes the question and answer as inputs and restricts the response to “Yes” or “No”.Source: §C.2 · p. 18

C.3 Prompt for multiple-choice questions with step-by-step reasoning

  • For multiple-choice questions requiring reasoning, the prompt requests step-by-step reasoning followed by a final choice in A, B, C, or D.Source: §C.3 · p. 19

Related papers

Loading 2510.10689v3…