Source-linked AI summary
Daily-Omni: Towards Audio-Visual Reasoning with Temporal Alignment across Modalities
Ziwei Zhou, Rui Wang, Zuxuan Wu, Yu-Gang Jiang
TL;DR
MLLMs have strong unimodal capabilities, but synchronous reasoning over temporally aligned audio and visual information remains insufficiently evaluated. Daily-Omni provides a 684-video, 1,197-question benchmark with scalable construction and diagnostic evaluation, finding that many end-to-end MLLMs struggle with fine-grained temporal alignment.
Problem
Existing MLLM benchmarks provide limited evidence about synchronous cross-modal reasoning, while audio-visual datasets often lack temporal alignment, broad domains, and rigorous evaluation.
Method
Daily-Omni combines 684 real-world videos, 1,197 multiple-choice questions across 6 task families, a scalable annotation and alignment pipeline, and a training-free modular diagnostic agent.
Results
Many end-to-end MLLMs struggle with fine-grained cross-modal temporal alignment, while the explicit-alignment Daily-Omni Agent provides a diagnostic reference with 61.82% overall accuracy.
Takeaways & Limitations
Daily-Omni identifies temporal alignment as a key bottleneck for current unified architectures and provides a benchmark for studying alignment-sensitive failures.
Abstract
from arXiv · showhide
Recent Multimodal Large Language Models (MLLMs) achieve promising performance on visual and audio benchmarks independently. However, the ability of these models to process cross-modal information synchronously remains largely unexplored. We introduce Daily-Omni, a multiple-choice Audio-Visual QA benchmark featuring 684 real-world videos and 1,197 questions spanning 6 task families that explicitly require cross-modal temporal reasoning. To support scalable benchmark construction, we develop a semi-automatic pipeline for annotation, cross-modal consistency refinement, temporal alignment elicitation, and text-only leakage filtering, followed by human verification. We further provide a diagnostic evaluation suite and extensively evaluate 24 foundation models under 37 model--modality settings (Audio+Video / Audio-only / Video-only / Text-only). Finally, we include a training-free modular diagnostic baseline that composes off-the-shelf unimodal models to serve as a diagnostic baseline and to illustrate how explicit temporal alignment signals affect performance. Results indicate that many end-to-end MLLMs still struggle on alignment-critical questions, suggesting that robust cross-modal temporal alignment remains an important open challenge.
1 Introduction
Daily-Omni addresses the lack of benchmarks for synchronously reasoning over temporally aligned audio and visual information. It introduces a scalable benchmark and diagnostic evaluation showing that many end-to-end MLLMs still struggle with deep audio-visual temporal integration.
- Motivation: Existing audio-visual benchmarks often use specialized domains, static image-audio pairs, narrow tasks, or insufficiently standardized evaluation.These limitations impede reliable assessment of temporal, cross-modal reasoning in everyday videos.
- Contributions: Daily-Omni comprises 684 real-life videos and 1,197 multiple-choice QA pairs across 6 major tasks involving audio-visual event alignment and cross-modal reasoning.
- Contributions: The automated QA pipeline covers annotation, revision, audio-visual temporal alignment, question generation, and optimization, with 30 hours of single-annotator quality filtering at approximately 30% acceptance.
- Contributions: The evaluation benchmarks 24 foundation models across 37 model–modality settings and includes modality ablations, alignment-sensitivity analyses, and a training-free modular reference baseline.
- Findings: Many end-to-end MLLMs struggle on deep audio-visual temporal integration, while the explicit-alignment baseline outperforms several recent open-source omni-modal models.
2 Related Works
Prior multimodal language-model and audio-visual benchmark research established unimodal capabilities but leaves important gaps in general, temporally dynamic audio-visual reasoning. Existing benchmarks vary in modality, scale, task type, scalability, domain coverage, and sound diversity.
- Multimodal Large Language Models: Recent MLLMs include audio, visual, and omni-modal language models, commonly using separate encoders for audio and visual inputs.
- Benchmark Comparison: Table 1 compares benchmarks by publication, modality, size, question type, automatic scalability, open-domain coverage, and general-sound diversity.
- Audio-Visual Benchmarks: Earlier audio-visual benchmarks often focus on specialized settings such as music performance or panoramic videos.
- Audio-Visual Benchmarks: Other benchmarks are restricted to short simple videos, static images, or narrowly emphasized audio tasks.
3 Daily-Omni
Daily-Omni is built from diverse real-world videos and a semi-automatic process that generates, revises, aligns, and verifies multimodal QA. Its diagnostic agent makes localized evidence explicit through modular, training-free processing.
- Dataset: Daily-Omni contains 684 real-world videos and 1,197 multiple-choice questions across 6 task families, using both 60-second and 30-second clips.
- Data Curation: The dataset emphasizes rich visual dynamics and everyday sounds beyond speech, while excluding largely static talking-head and non-English videos.
- QA Construction: The construction pipeline uses MLLMs for audio-visual annotation and revision, and reasoning LLMs for question, choice, and answer generation and optimization.
- Annotation and Revision: Audio and visual annotations are independently produced, visually consistency-checked, and cross-modally corrected to resolve sound misidentifications using visual context.
- Event Alignment: Event alignment identifies visual events occurring concurrently with each annotated audio event in the complete audio-visual clip.
- Question Types: Generated questions cover event alignment, event sequence, reasoning, inference, comparison, and context understanding.
- Diagnostic Baseline: The training-free Daily-Omni Agent combines Qwen2-Audio, Qwen2.5-VL-7B, Whisper-Large-V2, and Qwen2.5-14B-Instruct to process segmented audio, video, and speech evidence.
- Diagnostic Baseline: The agent uses targeted evidence selection because precise event-pair generation is difficult without a highly capable OLM and may overwhelm context or reasoning capacity.
4 Experiment
The experiments evaluate MLLMs across model types, modality settings, temporal alignment methods, and evaluation stability. Results show that explicit cross-modal alignment and access to both modalities are central to performance, while aggregate scores remain stable under subsampling.
- 4.2 Main Results: Earlier OLMs such as Unified-IO 2 and VideoLLaMA 2 show limited performance, sometimes falling below text-only LLMs.Unified-IO 2 also degrades with increasing model size under this evaluation setting.
- 4.2 Main Results: 61.82% overall accuracy is achieved by the training-free Daily-Omni Agent, while Gemini 2.5 Flash reaches up to 73.06%.The results reveal a performance divide among recent OLMs associated with their audio-visual temporal reasoning capabilities.
- 4.3 Evaluation Stability via Subsampling: At 80% video subsampling, the 5th–95th percentile range is within 1.1–1.2 percentage points for three representative models.The study uses 200 trials and approximately 958 questions at this sampling level.
- 4.4.1 Modality Ablation: Removing either modality substantially reduces accuracy: Gemini 2.5 Flash drops from 73.06% to 54.05% without vision and 44.61% without audio.Qwen3-Omni models likewise lose 13–16% when either modality is removed.
- 4.4.1 Modality Ablation: Audio-only performance is generally higher than visual-only performance, indicating that acoustic events and spoken dialogue are indispensable in Daily-Omni.This contrasts with video QA benchmarks where audio is often redundant.
- 4.4.2 Daily-Omni Agent Alignment: Smart Alignment achieves the highest average accuracy, whereas Naive Alignment slightly underperforms No Alignment because temporal grounding can produce erroneous event pairs.The study compares no alignment, question-agnostic naive alignment, and smart alignment, with aligned pairs restricted to sufficiently short segments.
5 Conclusion
Daily-Omni is an audio-visual QA benchmark and scalable construction pipeline for evaluating temporally aligned multimodal reasoning in daily-life scenarios. Evaluation across foundation models shows that fine-grained cross-modal temporal alignment remains a key bottleneck.
- 5 Conclusion: Daily-Omni evaluates temporally aligned multimodal reasoning through an audio-visual QA benchmark and scalable QA construction and verification pipeline.The benchmark targets daily-life scenarios.
- 5 Conclusion: Across 24 foundation models and 37 model–modality settings, recent MLLMs still struggle with questions requiring fine-grained cross-modal temporal alignment.The evaluation also includes the training-free Daily-Omni Agent as a modular reference baseline.
- 5 Conclusion: Daily-Omni identifies temporal alignment as a key bottleneck for current unified architectures.The paper points toward more accurate and robust multimodal temporal grounding and alignment mechanisms as future work.