Source-linked AI summary
LVOmniBench: Pioneering Long Audio-Video Understanding Evaluation for Omnimodal LLMs
Keda Tao, Yuhua Zheng, Jia Xu, Wenjie Du, Kele Shao, Hesong Wang, Xueyi Chen, Xin Jin, Junhan Zhu, Bohan Yu, Weiqiang Wang, Jian Liu, Can Qin, Yulun Zhang, Ming-Hsuan Yang, Huan Wang
TL;DR
Existing OmniLLM benchmarks largely emphasize short clips, leaving long-form integrated audio-visual understanding insufficiently evaluated. LVOmniBench addresses this gap with 275 long videos and 1,014 manually constructed QA pairs, finding substantial performance disparities: Gemini 3 Pro peaks at 65%, while open-source models remain below 35%.
Problem
Existing omnimodal benchmarks predominantly evaluate short clips or isolated modalities, limiting evidence about long-form integrated audio-visual understanding.
Method
LVOmniBench curates 275 diverse long videos and manually constructs 1,014 multiple-choice questions requiring joint audio-visual reasoning across hierarchical difficulty levels.
Results
Gemini 3 Pro peaks at 65% accuracy, whereas open-source models generally remain below 35%, revealing substantial challenges in long-form audio-visual understanding.
Takeaways & Limitations
LVOmniBench exposes a substantial proprietary–open-source performance disparity and highlights the need for improved joint processing of extended audio-visual sequences.
Takeaways & Limitations
Cross-modal semantic gaps remain a major limitation, with models often processing audio and video separately and struggling to align complementary information.
Abstract
from arXiv · showhide
Recent advancements in omnimodal large language models (OmniLLMs) have significantly improved the comprehension of audio and video inputs. However, current evaluations primarily focus on short audio and video clips ranging from 10 seconds to 5 minutes, failing to reflect the demands of real-world applications, where videos typically run for tens of minutes. To address this critical gap, we introduce LVOmniBench, a new benchmark designed specifically for the cross-modal comprehension of long-form audio and video. This dataset comprises high-quality videos sourced from open platforms that feature rich audio-visual dynamics. Through rigorous manual selection and annotation, LVOmniBench comprises 275 videos, ranging in duration from 10 to 90 minutes, and 1,014 question-answer (QA) pairs. LVOmniBench aims to rigorously evaluate the capabilities of OmniLLMs across domains, including long-term memory, temporal localization, fine-grained understanding, and multimodal perception. Our extensive evaluation reveals that current OmniLLMs encounter significant challenges when processing extended audio-visual inputs. Open-source models generally achieve accuracies below 35%, whereas the Gemini 3 Pro reaches a peak accuracy of approximately 65%. We anticipate that this dataset, along with our empirical findings, will stimulate further research and the development of advanced models capable of resolving complex cross-modal understanding problems within long-form audio-visual contexts.
1 Introduction
LVOmniBench is introduced to evaluate OmniLLMs on long-form, integrated audio-visual content through manually curated videos and annotations. Its results show that extended audio-visual processing remains difficult, with Gemini 3 Pro reaching 65% accuracy and open-source models below 35%.
- Dataset: 275 long videos across diverse scenarios comprise the dataset, totaling 140 hours and covering rich spatiotemporal and acoustic dynamics.
- Benchmark: LVOmniBench evaluates OmniLLMs’ cross-modal comprehension of long-form audio-visual content using strictly manual video curation and annotation.
- Dataset: Video durations range from 10 to 90 minutes, with an average duration of 2,069s and a greater than sixfold temporal-scale increase over existing audio-visual benchmarks.
- Evaluation: Questions span perception, understanding, inference, and complex logical reasoning, with three difficulty levels enabling hierarchical model evaluation.
- Results: 65% peak accuracy is achieved by Gemini 3 Pro, while open-source counterparts struggle to surpass 35% on long audio-visual sequences.
2 Related Work
Related work has progressed from single-modality perception toward omnimodal models that jointly process text, images, video, and audio. Existing unimodal benchmarks are relatively mature, but joint audio-visual evaluation remains challenging because prior benchmarks are shorter and automated question generation can miss complex reasoning requirements.
- Omnimodal MLLMs: Omnimodal architectures extend MLLMs from isolated single-modality perception to joint processing of text, images, video, and audio.Recent models are designed to process continuous video and audio streams and generate text and speech outputs.
- Evaluation Benchmarks: Unimodal benchmarks for image, video, and audio understanding are relatively mature, whereas joint audio-visual reasoning remains difficult to evaluate.The emergence of OmniLLMs has created new challenges for assessing cross-modal comprehension.
- Long-Form Evaluation: 30+ minutes is the duration of LVOmniBench videos, making them 6–20 times longer than those in previous benchmarks.The benchmark targets substantially longer-form audio-visual understanding than existing evaluations.
- Dataset Construction: All LVOmniBench videos and questions were manually selected and annotated by human experts to improve evaluation quality.The paper identifies hallucination and inadequate coverage of complex real-world reasoning as limitations of automated LLM-generated questions.
3 LVOmniBench
LVOmniBench is a long-form audio-visual benchmark built from dynamically selected, Creative Commons videos and designed to test cross-modal comprehension across perception, understanding, inference, and logical reasoning. Its final release contains 275 videos and 1,014 multiple-choice QA pairs spanning diverse categories and long-duration contexts.
- Corpus Construction: Creative Commons YouTube videos provide LVOmniBench with diverse, dynamically varying audio-visual content while preserving long-term accessibility and open-source availability.
- Corpus Construction: More than 3,000 raw videos were filtered into 275 high-quality videos lasting 10–90 minutes, each with a synchronized audio track.
- Question Taxonomy: The benchmark taxonomy evaluates multimodal perception, contextual understanding, spatiotemporal inference, and multi-step logical reasoning across long-form inputs.These dimensions cover fine-grained acoustic and visual features, entities and events, sound/spatial/temporal dynamics, and cross-modal reasoning chains.
- Annotation and Quality Control: Each video received 1–20 four-option multiple-choice questions, with question counts scaled to video duration and the density of relevant audio-visual events.
- Dataset Statistics: 1,014 multiple-choice questions remain after unimodal-baseline screening, yielding 275 videos across five categories and 21 subclasses with an average duration of 34 minutes and 29 seconds.The average duration is reported as 6–20 times longer than previous benchmarks, and the questions span nine categories.
4 Experimental Results
Experiments show that proprietary models substantially outperform open-source OmniLLMs on long-form audio-visual comprehension, while all models remain vulnerable to perception, localization, cross-modal alignment, and reasoning failures. Gemini 3.0 Pro achieves the strongest reported accuracy, whereas open-source models struggle especially on difficult tasks.
- Performance of Proprietary Models: 65.8%: Gemini 3.0 Pro achieves the highest overall accuracy, while Gemini 3.0 Flash reaches 59.0%.These results represent a 1.5-fold improvement over Gemini 2.0 Flash.
- Performance of Open-Source Models: 35.8%: Qwen3-Omni is the strongest open-source model, while every other open-source model remains below 35%.On high-difficulty tasks, open-source model performance degrades to near-random chance.
- Decisive Lead of Proprietary Models: Proprietary models consistently outperform open-source baselines across long-form audio-visual tasks and audio categories.The performance advantage is attributed to ultra-long context processing, audio comprehension, and temporal alignment.
- Impact of Audio Input: Raw audio improves Gemini 3.0 Flash over ASR text supplementation, indicating that non-verbal acoustic cues carry information beyond textual semantics.Adding ASR text to the visual stream does not improve accuracy over the original raw audio.
- Error Analysis: 34%: Perception errors are the largest reported error category, reflecting visual modality bias and difficulty interpreting fine-grained acoustic properties.Models often neglect corresponding audio signals while struggling with sound intensity, timbre, and pitch.
- Error Analysis: 23.5%, 19.6%, and 19%: reasoning, localization, and cross-modal semantic-gap errors remain major bottlenecks, alongside OCR and emotional-perception errors at 2% each.The failures involve temporal grounding, modality fusion, logical deduction, mathematical reasoning, event dynamics, and spatial relationships.
5 Conclusion
The paper introduces LVOmniBench, a benchmark for evaluating long audio-video understanding with diverse multimodal videos and manually annotated QA pairs. Evaluations reveal a substantial performance disparity between proprietary and open-source OmniLLMs, motivating architectures that jointly process extended audio and video sequences.
- LVOmniBench is a benchmark for long audio-video understanding featuring diverse videos with dynamic multimodal content.
- The benchmark uses manually annotated QA pairs to comprehensively evaluate OmniLLMs on complex joint audio-visual understanding.
- Evaluations reveal a substantial performance disparity between proprietary and open-source models.
- The findings underscore the need for advanced architectures capable of jointly processing extended audio and video sequences.
Supplementary Material · A Error Analysis Cases
The error analysis presents representative cases showing that OmniLLMs struggle with perception, localization, cross-modal semantics, and reasoning in long-form audio-visual understanding. The examples include a race-podium counting failure despite accurate audio-based event localization.
- A Error Analysis Cases: The section organizes representative failures across perception, localization, cross-modal semantic-gap, and reasoning errors.Figures 8–10 illustrate these error categories across audio-visual understanding tasks.
- A Error Analysis Cases: Fig. 9 illustrates localization errors in audio-visual understanding.The section identifies localization as a distinct challenge alongside perception and reasoning.
- A Error Analysis Cases: Fig. 10 demonstrates cross-modal semantic gaps and highlights reasoning errors.These cases concern failures that extend beyond isolated audio or visual perception.
- A Error Analysis Cases: Gemini3-Flash answered C when asked how many people stood on the third-place podium in the XTERRA off-road race.The stated error reason was a counting error in visual perception despite accurate audio localization.
- A Error Analysis Cases: 12:02 was the timestamp at which Gemini accurately located the XTERRA race event through audio.The model nevertheless failed on the associated visual counting question.
- A Error Analysis Cases: Fig. 8 presents perception errors caused by models misperceiving underlying visual or audio information.The figure covers failures in both audio and visual modalities.
B More Insights and Future Work
The section identifies persistent limitations in long-form audio-video understanding, including efficient processing, temporal localization, cross-modal alignment, and fine-grained audio perception. It frames these challenges as critical directions for future OmniLLM research.
- The Efficiency of Long-Context Capabilities: Efficient processing of joint long audio-video inputs remains a critical open problem because audio’s strict temporal continuity makes compression more difficult.Long contextual token sequences also complicate inference and increase computational overhead, while existing video-oriented strategies do not resolve the audio bottleneck.
- Time Grounding Capabilities and Benchmarks: Temporal localization in long-form audio-video requires significant improvement, with many errors arising from failures to locate events during initial inference.Audiovisual understanding additionally requires audio event localization and joint temporal alignment, beyond purely visual scene localization.
- Cross-Modal Alignment and Fine-Grained Understanding: Current OmniLLMs often over-rely on one modality and ignore information from the other, causing substantial cross-modal alignment errors.For complex problems, this modality bias hinders the integration of visual and audio features during reasoning.
- Abstract Audio Perception Capabilities: Models continue to make significant errors in fine-grained perception of audio frequency, timbre, emotional tone, and sound pressure levels.These non-linguistic attributes cannot be transcribed into text through ASR or alternative methods, motivating dedicated improvements in abstract audio perception.
D Evaluation Prompt
The benchmark evaluates both open-source and proprietary models with a standardized multiple-choice prompt requiring only the selected answer letter. Models must choose among provided options without reproducing option text or explaining their choice.
- The normal evaluation prompt is applied to both open-source and proprietary models.
- Each item presents a question and answer options, then asks the model to select the best answer.
- The required output is only the choice letter A/B/C/D, excluding the full option text and any explanation.