LVOmniBench: Pioneering Long Audio-Video Understanding Evaluation for Omnimodal LLMs

1 Introduction

• Long-form videos intertwine audio and visual streams over tens of minutes, complicating fine-grained understanding, cross-modal alignment, and reasoning.Existing omnimodal benchmarks mostly evaluate static image-audio pairs or short-form video clips, leaving a gap in long-form joint audio-video evaluation.Source: §1 · p. 2
• LVOmniBench manually curates 275 diverse videos totaling 140 hours (10–90 minutes each; 2,069 seconds on average) and 1,014 high-quality multiple-choice questions.The average video duration represents a greater than sixfold increase in temporal scale over existing audio-visual benchmarks.Source: §1 · p. 3
• Questions require joint audio-visual reasoning and cover perception, understanding, inference, and complex logical reasoning across three difficulty levels.The question types and difficulty tiers support hierarchical evaluation of model performance.Source: §1 · p. 3
• Gemini 3 Pro reaches only 65% accuracy, while open-source models struggle to surpass 35% on long audio-visual sequences.The findings underscore the difficulty of processing extended inputs and achieving cross-modal alignment.Source: §1 · p. 4

2.1 Omnimodal Large Language Models

• Omnimodal research is moving beyond isolated perception toward continuous audio-video processing, yet models still struggle with long-range temporal reasoning [1] [13] [15] [30] [46] [54] [72] [75] [82] [84] [94] [95] [98] [102] [103].Source: §2.1 · p. 4

2.2 Multimodal Benchmarks

• Most joint audio-visual benchmarks remain domain-specific or short-clip-focused, while OmniVideoBench contains only a subset of 10–30-minute videos and mostly clips lasting a few minutes [43].Source: §2.2 · p. 4
• LVOmniBench videos average over 30 minutes—six to twenty times longer than prior benchmarks—and human experts manually selected and annotated all videos and questions.Automated LLM question generation can miss complex real-world reasoning and remains prone to hallucination [6] [7] [115].Source: §2.2 · p. 5

3.1 Video Collection

• The collection uses Creative Commons YouTube videos screened across five domains and 21 fine-grained subcategories for dynamic, informative audio-visual content.From more than 3,000 initial videos, the authors selected 275 high-quality videos with synchronized audio, each 10–90 minutes long; most run 20–45 minutes.Source: §3.1 · p. 5

3.2 Question Answer Annotation

• The taxonomy evaluates perception, entity and action understanding, spatiotemporal sound inference, and multi-step logical reasoning grounded in complementary audio-visual cues.Its subcategories include counting, attribute and music perception; human-centric and event understanding; sound, spatial and temporal inference; and logical tasks requiring extended contextual dependencies.Source: §3.2 · pp. 6–7

3.4 Dataset Statistics and Comparison

• Speech, music, and sound answer 763, 137, and 114 questions, respectively; low, medium, and high difficulty tiers contain 315, 441, and 259 questions.All three difficulty tiers are represented across every question category.Source: Figure 4 · p. 7, §3.4 · p. 8

3.2 Question Answer Annotation

• Annotators created one to 20 four-option questions per video, requiring unambiguous joint audio-visual reasoning and avoiding commonsense shortcuts, unequal options, or timestamp clues.The initial round produced more than 1,500 candidate QA pairs; question counts varied with video duration and relevant event density.Source: §3.2 · p. 7

3.3 Question Refine and Filtering

• Unimodal baseline tests and reasoning-summary reviews guided revisions or removals of questions answerable from one modality, common sense, timestamps, or explicit descriptions; the final set has 1,014 QA pairs.Each pair was rated for perceptual difficulty, information granularity, temporal span, and inference complexity, then assigned low, medium, or high difficulty.Source: §3.3 · pp. 7–8

3.4 Dataset Statistics and Comparison

• 275 videos averaging 34 minutes 29 seconds and 1,014 multiple-choice questions across nine categories make LVOmniBench 6–20 times longer than previous benchmarks [6] [7] [35] [43] [47] [74] [115].The questions average 16.4 words.Source: Table 2 · p. 8, §3.4 · p. 8

4.1 Experimental Settings

• The evaluation covered open-source OmniLLMs, specialized video and audio baselines, and Gemini, using maximum-permissible frames; selected models received 768, 64, or 16 frames under context limits.Local experiments used official model configurations.Source: §4.1 · pp. 8–9

4.2 Quantitative Results

• 65.8% accuracy (Gemini 3.0 Pro) and 59.0% (Flash) led LVOmniBench, about 1.5-fold above Gemini 2.0 Flash; Qwen3-Omni led open models at 35.8%, others below 35%.Open-source model performance on high-difficulty tasks degraded to near-random chance.Source: Table 3 · p. 9, §4.2 · pp. 9–10
• 45% accuracy on high-difficulty questions for Gemini 3.0 Pro accompanied persistent challenges in counting and music perception, especially for non-verbal and abstract music sounds.The paper also reports that open-source models approach random-chance performance on high-difficulty tasks.Source: Figure 5 · p. 10, §4.2 · p. 10, Figure 6 · p. 11

4.3 Key Takeaways

• ASR transcripts improved Qwen3-Omni and Ming-Flash-Omni-2.0, but not Gemini 3.0 Flash over raw audio, which retains non-verbal acoustic cues.The authors interpret this contrast as evidence that open-source models struggle to use extended audio information effectively.Source: Table 4 · p. 12, §4.3 · pp. 12–13
• As input duration increases, performance degrades relative to short-video benchmarks; dense, sequential long audio is identified as a key modeling bottleneck.The paper calls for stronger long-form audio-visual modeling in future OmniLLMs.Source: §4.3 · p. 13

4.4 Error Analysis

• 34% of analyzed Gemini 3 Flash errors were perception errors, split between audio and visual failures; models often neglected audio cues or struggled with fine-grained acoustic and visual details.The figure reports that 60% of perception errors were audio-related and 40% visual-related.Source: §4.4 · pp. 13–14, Figure 7 · p. 13
• Reasoning errors accounted for 23.5%, localization errors 19.6%, and cross-modal semantic-gap errors 19%; OCR and emotional-perception errors each accounted for 2%.The errors included difficulty with deduction, temporal grounding, and aligning complementary audio-visual information.Source: §4.4 · p. 14

5 Conclusion

• LVOmniBench introduces diverse long-form videos with dynamic multimodal content and manually annotated QA pairs to evaluate OmniLLMs on complex joint audio-visual understanding.Source: §5 · pp. 14–15
• Evaluations reveal a substantial performance disparity between proprietary and open-source models, highlighting the need for architectures that jointly process extended audio and video.Source: §5 · p. 15
• The authors hope LVOmniBench will catalyze future research and drive the development of more robust and comprehensive OmniLLMs.Source: §5 · p. 15

A Error Analysis Cases

• The examples span perception, localization, cross-modal semantic-gap, and reasoning errors in audio-visual understanding.Figures 8, 9, and 10 respectively illustrate perception errors, localization errors, and the cross-modal semantic gap alongside reasoning errors.Source: A Error Analysis Cases · p. 22
• Fig. 9 illustrates localization errors.Source: A Error Analysis Cases · p. 22
• Fig. 10 demonstrates a cross-modal semantic gap and reasoning errors.Source: A Error Analysis Cases · p. 22
• Addressing these errors requires better long-form audio-visual comprehension, from basic perception and localization to advanced reasoning and cross-modal understanding.Source: A Error Analysis Cases · p. 22
• The cases indicate substantial room for improvement, and LVOmniBench aims to guide future research.Source: A Error Analysis Cases · p. 22
• Fig. 8 shows perception failures caused by models misperceiving underlying visual or audio information.The cases cover both audio and visual modalities.Source: Figure 8 · p. 22, A Error Analysis Cases · p. 22

The Efficiency of Long-Context Capabilities

• Audio remains harder to compress than video because of its strict temporal continuity, leaving efficient processing of joint long audio-video inputs an open problem.Long contextual token sequences also complicate inference and incur substantial computational overhead.Source: B More Insights and Future Work · pp. 22–23

Time Grounding Capabilities and Benchmarks

• Most temporal-localization errors arise during initial inference, and models struggle to locate events in long-form audio and video inputs.Effective audiovisual understanding requires audio-event localization and joint temporal alignment in addition to visual scene localization; comprehensive benchmarks and methods remain lacking.Source: B More Insights and Future Work · p. 23, Figure 9 · p. 23

Cross-Modal Alignment and Fine-Grained Understanding

• OmniLLMs often over-rely on one modality, causing cross-modal alignment errors and making fine-grained audio-visual comprehension difficult.Training audio and visual modalities for alignment remains complex.Source: B More Insights and Future Work · p. 23, Figure 10 · p. 24

Abstract Audio Perception Capabilities

• Models make significant errors perceiving audio frequency of occurrence, timbre, emotional tone, and sound pressure levels, motivating better comprehension of raw audio signals.These non-linguistic attributes cannot be transcribed into text for comprehension via ASR or alternative methods.Source: B More Insights and Future Work · p. 23

• The same prompt is used for normal evaluation of open-source and proprietary models.Source: D Evaluation Prompt · p. 24
• Models are asked to select the best answer from the question’s provided options.Source: D Evaluation Prompt · p. 24
• Responses must contain only the choice letter (A/B/C/D), with no option text or explanation.Source: D Evaluation Prompt · p. 24