PaperlayerView exact revision on arXiv

Source-linked AI summary

Daily-Omni: Towards Audio-Visual Reasoning with Temporal Alignment across Modalities

Ziwei Zhou, Rui Wang, Zuxuan Wu, Yu-Gang Jiang

arXiv:2505.17862v2cs.AIcs.CLcs.CV

TL;DR

MLLMs have strong unimodal capabilities, but synchronous reasoning over temporally aligned audio and visual information remains insufficiently evaluated. Daily-Omni provides a 684-video, 1,197-question benchmark with scalable construction and diagnostic evaluation, finding that many end-to-end MLLMs struggle with fine-grained temporal alignment.

  • Problem

    Existing MLLM benchmarks provide limited evidence about synchronous cross-modal reasoning, while audio-visual datasets often lack temporal alignment, broad domains, and rigorous evaluation.

    Source: §1 · p. 2

  • Method

    Daily-Omni combines 684 real-world videos, 1,197 multiple-choice questions across 6 task families, a scalable annotation and alignment pipeline, and a training-free modular diagnostic agent.

    Source: §1 · pp. 2–3, §3.2 · p. 5, §3.3 · p. 6

  • Results

    Many end-to-end MLLMs struggle with fine-grained cross-modal temporal alignment, while the explicit-alignment Daily-Omni Agent provides a diagnostic reference with 61.82% overall accuracy.

    Source: §1 · p. 3, §4.2 · p. 8

  • Takeaways & Limitations

    Daily-Omni identifies temporal alignment as a key bottleneck for current unified architectures and provides a benchmark for studying alignment-sensitive failures.

    Source: §5 · p. 11

Abstract

from arXiv · show

Recent Multimodal Large Language Models (MLLMs) achieve promising performance on visual and audio benchmarks independently. However, the ability of these models to process cross-modal information synchronously remains largely unexplored. We introduce Daily-Omni, a multiple-choice Audio-Visual QA benchmark featuring 684 real-world videos and 1,197 questions spanning 6 task families that explicitly require cross-modal temporal reasoning. To support scalable benchmark construction, we develop a semi-automatic pipeline for annotation, cross-modal consistency refinement, temporal alignment elicitation, and text-only leakage filtering, followed by human verification. We further provide a diagnostic evaluation suite and extensively evaluate 24 foundation models under 37 model--modality settings (Audio+Video / Audio-only / Video-only / Text-only). Finally, we include a training-free modular diagnostic baseline that composes off-the-shelf unimodal models to serve as a diagnostic baseline and to illustrate how explicit temporal alignment signals affect performance. Results indicate that many end-to-end MLLMs still struggle on alignment-critical questions, suggesting that robust cross-modal temporal alignment remains an important open challenge.

1 Introduction

Daily-Omni addresses the lack of benchmarks for synchronously reasoning over temporally aligned audio and visual information. It introduces a scalable benchmark and diagnostic evaluation showing that many end-to-end MLLMs still struggle with deep audio-visual temporal integration.

Motivation

  • Existing audio-visual benchmarks often use specialized domains, static image-audio pairs, narrow tasks, or insufficiently standardized evaluation.These limitations impede reliable assessment of temporal, cross-modal reasoning in everyday videos.Source: §1 · p. 2

Contributions

  • Daily-Omni comprises 684 real-life videos and 1,197 multiple-choice QA pairs across 6 major tasks involving audio-visual event alignment and cross-modal reasoning.Source: §1 · p. 2
  • The automated QA pipeline covers annotation, revision, audio-visual temporal alignment, question generation, and optimization, with 30 hours of single-annotator quality filtering at approximately 30% acceptance.Source: §1 · p. 2
  • The evaluation benchmarks 24 foundation models across 37 model–modality settings and includes modality ablations, alignment-sensitivity analyses, and a training-free modular reference baseline.Source: §1 · p. 3

Findings

  • Many end-to-end MLLMs struggle on deep audio-visual temporal integration, while the explicit-alignment baseline outperforms several recent open-source omni-modal models.Source: §1 · p. 3

2 Related Works

Prior multimodal language-model and audio-visual benchmark research established unimodal capabilities but leaves important gaps in general, temporally dynamic audio-visual reasoning. Existing benchmarks vary in modality, scale, task type, scalability, domain coverage, and sound diversity.

Multimodal Large Language Models

  • Recent MLLMs include audio, visual, and omni-modal language models, commonly using separate encoders for audio and visual inputs.Source: §2.1 · p. 3

Benchmark Comparison

  • Table 1 compares benchmarks by publication, modality, size, question type, automatic scalability, open-domain coverage, and general-sound diversity.Source: Table 1 · p. 3

Audio-Visual Benchmarks

  • Earlier audio-visual benchmarks often focus on specialized settings such as music performance or panoramic videos.Source: §2.2 · p. 4
  • Other benchmarks are restricted to short simple videos, static images, or narrowly emphasized audio tasks.Source: §2.2 · p. 4

3 Daily-Omni

Daily-Omni is built from diverse real-world videos and a semi-automatic process that generates, revises, aligns, and verifies multimodal QA. Its diagnostic agent makes localized evidence explicit through modular, training-free processing.

Dataset

  • Daily-Omni contains 684 real-world videos and 1,197 multiple-choice questions across 6 task families, using both 60-second and 30-second clips.Source: §3 · p. 4

Data Curation

  • The dataset emphasizes rich visual dynamics and everyday sounds beyond speech, while excluding largely static talking-head and non-English videos.Source: §3.1 · p. 4

QA Construction

  • The construction pipeline uses MLLMs for audio-visual annotation and revision, and reasoning LLMs for question, choice, and answer generation and optimization.Source: §3.2 · p. 4

Annotation and Revision

  • Audio and visual annotations are independently produced, visually consistency-checked, and cross-modally corrected to resolve sound misidentifications using visual context.Source: §3.2 · p. 5

Event Alignment

  • Event alignment identifies visual events occurring concurrently with each annotated audio event in the complete audio-visual clip.Source: §3.2 · p. 5

Question Types

  • Generated questions cover event alignment, event sequence, reasoning, inference, comparison, and context understanding.Source: §3.2 · pp. 5–6

Diagnostic Baseline

  • The training-free Daily-Omni Agent combines Qwen2-Audio, Qwen2.5-VL-7B, Whisper-Large-V2, and Qwen2.5-14B-Instruct to process segmented audio, video, and speech evidence.Source: §3.3 · pp. 6–7
  • The agent uses targeted evidence selection because precise event-pair generation is difficult without a highly capable OLM and may overwhelm context or reasoning capacity.Source: §3.3 · p. 7

4 Experiment

The experiments evaluate MLLMs across model types, modality settings, temporal alignment methods, and evaluation stability. Results show that explicit cross-modal alignment and access to both modalities are central to performance, while aggregate scores remain stable under subsampling.

4.2 Main Results

  • Earlier OLMs such as Unified-IO 2 and VideoLLaMA 2 show limited performance, sometimes falling below text-only LLMs.Unified-IO 2 also degrades with increasing model size under this evaluation setting.Source: §4.2 · p. 8
  • 61.82% overall accuracy is achieved by the training-free Daily-Omni Agent, while Gemini 2.5 Flash reaches up to 73.06%.The results reveal a performance divide among recent OLMs associated with their audio-visual temporal reasoning capabilities.Source: §4.2 · p. 8, §4.4.2 · p. 10

4.3 Evaluation Stability via Subsampling

  • At 80% video subsampling, the 5th–95th percentile range is within 1.1–1.2 percentage points for three representative models.The study uses 200 trials and approximately 958 questions at this sampling level.Source: §4.3 · p. 8, Figure 6 · p. 10

4.4.1 Modality Ablation

  • Removing either modality substantially reduces accuracy: Gemini 2.5 Flash drops from 73.06% to 54.05% without vision and 44.61% without audio.Qwen3-Omni models likewise lose 13–16% when either modality is removed.Source: §4.4.1 · p. 10
  • Audio-only performance is generally higher than visual-only performance, indicating that acoustic events and spoken dialogue are indispensable in Daily-Omni.This contrasts with video QA benchmarks where audio is often redundant.Source: §4.4.1 · p. 10

4.4.2 Daily-Omni Agent Alignment

  • Smart Alignment achieves the highest average accuracy, whereas Naive Alignment slightly underperforms No Alignment because temporal grounding can produce erroneous event pairs.The study compares no alignment, question-agnostic naive alignment, and smart alignment, with aligned pairs restricted to sufficiently short segments.Source: §4.4.2 · pp. 10–11

5 Conclusion

Daily-Omni is an audio-visual QA benchmark and scalable construction pipeline for evaluating temporally aligned multimodal reasoning in daily-life scenarios. Evaluation across foundation models shows that fine-grained cross-modal temporal alignment remains a key bottleneck.

5 Conclusion

  • Daily-Omni evaluates temporally aligned multimodal reasoning through an audio-visual QA benchmark and scalable QA construction and verification pipeline.The benchmark targets daily-life scenarios.Source: §5 · p. 11
  • Across 24 foundation models and 37 model–modality settings, recent MLLMs still struggle with questions requiring fine-grained cross-modal temporal alignment.The evaluation also includes the training-free Daily-Omni Agent as a modular reference baseline.Source: §5 · p. 11
  • Daily-Omni identifies temporal alignment as a key bottleneck for current unified architectures.The paper points toward more accurate and robust multimodal temporal grounding and alignment mechanisms as future work.Source: §5 · p. 11

Related papers

Loading 2505.17862v2…