Source-linked AI summary
OmniVideoBench: Towards Audio-Visual Understanding Evaluation for Omni MLLMs
Caorui Li, Yu Chen, Yiyan Ji, Jin Xu, Zhenyu Cui, Shihao Li, Yuanxing Zhang, Wentao Wang, Zhenghao Song, Dingling Zhang, Ying He, Haoxiang Liu, Yuxuan Wang, Qiufeng Wang, Jiafu Tang, Zhenhe Wu, Jiehui Luo, Zhiyu Pan, Weihao Xie, Chenchen Zhang, Zhaohui Wang, Jiayi Tian, Yanghai Wang, Zhe Cao, Minxin Dai, Ke Wang, Runzhe Wen, Yinghao Ma, Yaning Pan, Sungkyun Chang, Termeh Taheri, Haiwen Xia, Christos Plachouras, Emmanouil Benetos, Yizhi Li, Ge Zhang, Jian Yang, Tianhao Peng, Zili Wang, Minghao Liu, Junran Peng, Zhaoxiang Zhang, Jiaheng Liu
TL;DR
Existing benchmarks underdevelop audio-visual reasoning evaluation by often emphasizing short clips or a single modality. OmniVideoBench introduces a diverse, reasoning-annotated benchmark, and evaluations show that models remain substantially challenged, particularly on long videos and music-related audio.
Problem
Existing benchmarks often focus on short clips or a single modality, leaving long-term audio-visual reasoning underdeveloped.
Method
OmniVideoBench evaluates audio-visual collaborative reasoning using 628 diverse videos, 1,000 QA pairs, and explicit step-by-step reasoning chains.
Results
Models remain below a 60% passing score overall, while performance is especially weak on long videos and music-related audio.
Takeaways & Limitations
OmniVideoBench exposes gaps in modality complementarity, long-form temporal reasoning, and music understanding relative to human-level performance.
Abstract
from arXiv · showhide
Recent advances in multimodal large language models (MLLMs) have demonstrated substantial potential in video understanding. However, existing benchmarks fail to comprehensively evaluate synergistic reasoning capabilities across audio and visual modalities, often neglecting either one of the modalities or integrating them in a logically inconsistent manner. To bridge this gap, we introduce OmniVideoBench, a large-scale and rigorously designed benchmark dedicated to assessing synergistic audio-visual understanding, with a strong emphasis on modality complementarity and logical consistency. Specifically, OmniVideoBench comprises 1000 high-quality question-answer(QA) pairs, each annotated with step-by-step reasoning traces, derived from 628 diverse videos ranging from several seconds to 30 minutes, and manually verified to guarantee complete correctness and uniqueness. Moreover, OmniVideoBench encompasses 13 carefully designed question types, covering temporal reasoning, spatial localization, counting, causal inference, summarization, and beyond, thereby capturing the essential challenges of video understanding. Evaluation of multiple MLLMs on OmniVideoBench reveals a pronounced gap between model performance and human reasoning, with open-source models lagging significantly behind their closed-source counterparts, underscoring the inherent difficulty of genuine audio-visual reasoning. We will release OmniVideoBench to foster the development of MLLMs with stronger and more generalizable reasoning capabilities.
NJU-LINK Team
The full author list appears in the Contributions section.
- The Contributions section contains the full author list.
1 Introduction
Existing benchmarks incompletely evaluate audio-visual reasoning, especially over long, multimodal videos. OmniVideoBench addresses this gap with diverse videos, reasoning-chain annotations, and evaluations revealing substantial performance weaknesses across models, audio conditions, and task types.
- Existing benchmarks often emphasize short clips or a single modality, limiting evaluation of long-term audio-visual reasoning.
- OmniVideoBench uses diverse videos and 1,000 question-answer pairs with step-by-step reasoning chains that identify modality and evidence.
- 58.90% is the best reported accuracy, and current models do not achieve a 60% passing score on OmniVideoBench.
- Long-video understanding remains difficult for models including Gemini-2.0-Flash and Qwen3-Omni-30B-A3B, despite stronger performance from some leading systems.
- Gemini-2.5-Pro reaches 38.46% on music videos versus 57.72% on sound and 61.66% on speech, showing strong variation by audio signal.
- Gemini-2.5-Pro exceeds 80% on relationship reasoning and summarization but remains below 50% on background and music understanding.
2 OmniVideoBench
OmniVideoBench evaluates audio-visual collaborative reasoning through diverse real-world videos, broad task coverage, and manually refined questions with explicit multimodal reasoning chains. Its construction emphasizes modality complementarity, balanced answer choices, and filtering of questions solvable without genuine video reasoning.
- Overview: OmniVideoBench evaluates MLLMs’ ability to synthesize audio, visual, and textual information into explicit reasoning steps.The benchmark’s main task requires processing video, audio, and associated text to produce answers supported by reasoning.
- Video Collection: The benchmark uses real-world videos selected for diverse types, durations, and complementary audio-visual content.Collection criteria also restrict some sources to recent publications to reduce overlap with existing training data.
- Question Design: Questions span 13 task types, including perception, spatial and temporal reasoning, counting, summarization, causal reasoning, and hypothetical reasoning.The initial annotation process produced approximately 2,500 multiple-choice QA pairs before filtering and refinement.
- Question Design: Answer choices are designed to limit textual shortcuts through consistent formatting, relevant distractors, and balanced semantic distances.Semantic distance is defined using the symmetric difference between sets of semantic units, and distractors are required to have consistent distances from one another and the correct answer.
- Quality Assurance: Unimodal, text-solvable, incorrect, non-unique, and mismatched questions are filtered through model-assisted screening and manual refinement.The final annotation adds atomic reasoning steps containing modality, evidence, and inference, yielding 1,000 high-quality QA pairs.
- Dataset Statistics: The resulting dataset contains 628 videos across 8 major and 68 fine-grained categories, with 1,000 QA pairs spanning 13 task types.Figure 3 further summarizes video-duration intervals, task distributions, and speech, sound, and music coverage.
3 Experiments
Experiments show substantial gaps in omni-modal video reasoning, with closed-source models generally outperforming open-source systems and performance varying sharply by task, audio type, and video length.
- Main Results: Most MLLMs still struggle with long videos, although some leading systems such as Gemini-2.5-Pro show relatively robust performance.Gemini-2.0-Flash and Qwen3-Omni remain examples of models with difficulty on long-video understanding.
- Main Results: Gemini-2.5-Pro achieves the highest accuracy on 11 of 13 tasks, while closed-source models generally outperform open-source models across task types.Its strongest results occur in relationship, spatial, referential, and cause-and-effect reasoning.
- Main Results: Current MLLMs struggle with music-related audio, and Background and Music Understanding remains challenging even for Gemini-2.5-Pro, which scores below 50%.Music requires connecting low-semantic acoustic cues such as musical style and tempo changes with higher-level reasoning.
- Further Analysis: ASR transcripts improve visual-only MLLMs, but direct audio-visual processing can still underperform text-assisted visual reasoning, revealing limited cross-modal integration.Qwen2.5-Omni-7B performs worse than the similarly sized Qwen2.5-VL-7B despite processing audio and visual inputs together.
- Further Analysis: Increasing sampled frames from 32 to 256 steadily improves accuracy, with larger gains for longer videos.The authors attribute this pattern to richer temporal coverage, more complete motion cues, and stronger cross-modal alignment.
- Further Analysis: Open-ended QA reduces model accuracy compared with multiple-choice evaluation, removing guessing and answer-option lexical cues.Table 4 reports the comparison across various models.
4 Related Works
Related work traces MLLMs from vision-language integration toward omni-modal systems and increasingly capable benchmarks that probe deeper cognitive skills.
- Omni-Understanding MLLMs: Omni-modal MLLMs extend earlier vision-language systems toward processing and generating information across arbitrary combinations of modalities.This paradigm treats the language model as a central engine for unifying diverse data.
- MLLM Benchmarks: MLLM benchmarks have progressed from foundational perception evaluation toward frameworks assessing more sophisticated cognitive abilities.Examples include MLLM-Bench, MMMU, and MMMU-Pro.
5 Conclusion
The paper introduces OmniVideoBench to evaluate collaborative audio-visual reasoning with diverse videos, verified questions, and explicit reasoning annotations.
- Conclusion: OmniVideoBench targets audio-visual collaborative reasoning using diverse videos, carefully verified QA pairs, and explicit reasoning annotations.The benchmark is intended to support stronger and more generalizable multimodal reasoning systems.
- Conclusion: Experiments show that both open- and closed-source models struggle with modality complementarity, long-form temporal reasoning, and music understanding relative to human-level performance.These findings identify persistent challenges for current omni-modal models.
6 Contributions
The supplied contribution entries identify team work on annotation management and data quality inspection.
- Contributions: The contribution list includes data annotation management as a team responsibility.
- Contributions: The contribution list includes data quality inspection as a team responsibility.
- Contributions: The listed contributors are affiliated with Alibaba Group, Southeast University, Nanjing University, Kuaishou Technology, M-A-P, and the University of Science and Technology Beijing.
A Full Video Category Taxonomy
OmniVideoBench organizes its video dataset into a broad taxonomy spanning 8 major categories and 68 subcategories.
- 8 major categories and 68 subcategories define the benchmark’s full video taxonomy.
B Detailed Principles of Video Collection
Video collection prioritizes diverse, information-rich content in which audio and visual streams are complementary and independently meaningful. Selection also filters for dynamic visuals, aligned audio, unobstructed presentation, recent publication, and sufficient resolution.
- Videos must provide rich information across audio and visual modalities while preserving complementarity between the streams.
- Videos are restricted to those published after June 2024 to reduce exposure-related unfairness and potential performance overestimation.
- The collection excludes videos lacking sufficient dynamic visual information for meaningful video-understanding evaluation.
- Audio must align with the visual content, whether it consists of speech, environmental sound, or music.
- Embedded subtitles and large text overlays are excluded because they can visually reveal audio content or ongoing events.
- Videos require at least 480p resolution and must avoid distortion or blurriness that hinders comprehension.
C Prompts Used in This Work
The work uses prompts for direct multiple-choice answering, option-dependence filtering, and step-by-step reasoning. These prompts specify strict output formats and distinguish questions answerable without options from those requiring them.
- C.1 Prompt for Overall Evaluation: Overall evaluation asks the model to answer a video-based multiple-choice question with only the selected option letter.
- C.1 Prompt for Overall Evaluation: When video access is limited, the evaluation prompt still requires choosing the option judged more likely than the alternatives.
- C.2 Prompt to select questions that can be answered without relying on options: An impartial judge determines whether each question inherently depends on its multiple-choice options, assuming full knowledge of the video.
- C.2 Prompt to select questions that can be answered without relying on options: Questions answerable without options are labeled “No,” even when their exact wording might change slightly.
- C.2 Prompt to select questions that can be answered without relying on options: Questions that cannot be answered without options, including explicit “Which of the following...” questions, are labeled “Yes.”
- C.2 Prompt to select questions that can be answered without relying on options: The option-dependence judge must respond only with “Yes” or “No,” using the supplied question and answer fields.
- C.3 Prompt for multiple-choice questions with step-by-step reasoning: The reasoning prompt requires step-by-step justification followed by a final answer formatted as the chosen option letter.