PaperlayerView exact revision on arXiv

Source-linked AI summary

Audio-Visual Flamingo: Open Audio-Visual Intelligence for Long and Complex Videos

Sreyan Ghosh, Arushi Goel, Kaousheik Jayakumar, Lasha Koroshinadze, Nishit Anand, Siddharth Gururani, Hanrong Ye, Pritam Biswas, Yuanhang Su, Ehsan Hosseini-Asl, Sang-gil Lee, Zhifeng Kong, Jaehyeon Kim, Sungwon Kim, S Sakshi, Ramani Duraiswami, Dinesh Manocha, Andrew Tao, Mohammad Shoeybi, Bryan Catanzaro, Ming-Yu Liu, Wei Ping

arXiv:2607.16107v1eess.AScs.CV

TL;DR

Long, complex real-world videos require joint audio-visual understanding and reasoning, but existing resources and models provide limited support for this setting. AV-Flamingo addresses this with cross-modal data, staged training, and timestamp-grounded reasoning, achieving strong performance across diverse benchmarks, especially on long and complex videos.

  • Problem

    Joint audio-visual understanding and cross-modal reasoning for long, complex real-world videos remain limited despite their rich, continuous visual and auditory content.

  • Method

    AV-Flamingo combines a cross-modal audio-visual architecture, the AV-Skills dataset, a three-stage curriculum, and timestamp-grounded reasoning for long videos.

  • Results

    AV-Flamingo outperforms similar-sized audio-visual and omni-modal models across 15+ benchmarks and remains competitive with larger open-weight and closed models.

  • Takeaways & Limitations

    AV-Flamingo demonstrates strong performance across diverse benchmarks, with especially notable gains on long and complex real-world videos.

  • Takeaways & Limitations

    AV-Flamingo may be affected by source bias and training-data overlap, while very long dense videos and open-ended real-world deployment remain challenging.

Abstract

from arXiv · show

We present Audio-Visual Flamingo (AV-Flamingo), a fully open state-of-the-art audio-visual large language model (AV-LLM) for joint understanding and reasoning over audio, images, and long-form videos. Unlike prior AV-LLMs that primarily focus on short clips, AV-Flamingo is designed for understanding and reasoning over long and complex real-world (audio-visual) videos. To support this, we make three key contributions: (i) Audio-Visual-Skills, a large-scale collection of real-world videos with ~7M caption and question-answer training instances designed to emphasize temporal, compositional, and cross-modal audio-visual reasoning; (ii) a novel three-stage curriculum that progressively trains the model from short-range perception to long-horizon multi-event reasoning; and (iii) Temporal Audio-Visual Interleaved Chain-of-Thought, a reasoning framework that explicitly grounds intermediate reasoning steps to timestamps in long audio-visual streams, improving temporal alignment and interpretability. Extensive experiments across 15+ audio-visual, omni-modal, audio, and vision benchmarks show that AV-Flamingo outperforms similarly sized open models by clear margins and remains highly competitive with, and in some cases surpasses, much larger open-weight and closed models, particularly on long and complex real-world audio-visual understanding and reasoning tasks. Beyond benchmark performance, AV-Flamingo exhibits strong real-world utility and transfers well to unseen tasks, highlighting its robustness and generalization ability.

1. Introduction

AV-Flamingo addresses the gap in long-form audio-visual understanding, where existing models often neglect audio, focus on short videos, and lack suitable jointly annotated data. It introduces a fully open model and scalable training recipe for temporally grounded reasoning over long, complex real-world videos, achieving strong benchmark and robustness results.

  • Motivation: Long, information-rich videos combine visuals, speech, sounds, and music, but current models often fail to process these modalities jointly.Audio is described as a fundamental component of the viewing experience.
  • Problem: Most recent audio-visual and omni-modal LLMs focus on short videos, despite widespread consumption of medium- and long-form content such as movies, lectures, and documentaries.The paper identifies this mismatch as a central limitation of existing video understanding systems.
  • Contributions: AV-Flamingo is a fully open AV-LLM for joint audio-visual understanding and reasoning over long and complex real-world videos.The model is presented as a first step toward scaling open audio-visual intelligence beyond academic benchmarks.
  • Contributions: The proposed recipe combines internet-scale audio-visual data curation, targeted capability expansion, and temporally grounded reasoning for long videos.These components are designed to address the scarcity of suitable long-form audio-visual data and weaknesses in joint AV perception.
  • Results: 15+ benchmarks show AV-Flamingo outperforming similar-sized AV and omni-modal LLMs while remaining competitive with, and sometimes surpassing, much larger models.The paper also reports substantially stronger robustness on long and complex real-world videos.

2. Related Works

Related work has advanced vision-language and audio-language models, but unified audio-visual modeling remains limited. Long-form video understanding and high-quality joint audio-visual supervision are persistent open challenges.

  • Audio-Visual Large Language Models: Vision-language models have progressed from Flamingo, BLIP-2, and LLaVA to newer systems including InternVL 2.5, Qwen3-VL, VideoLLaMA 3, and Qwen3.5.
  • Audio-Visual Large Language Models: Audio-language models such as LTU, SALMONN, Qwen2-Audio, and Audio Flamingo 3 have advanced speech, sound, and music understanding.
  • Long-Form Video Understanding and Temporal Reasoning: Most current video LLMs remain limited to clips shorter than one minute, while long-form efforts often focus on vision-only settings without audio.
  • Long-Form Video Understanding and Temporal Reasoning: Many long-video approaches rely on context reduction, which can hurt performance in real-world scenarios.
  • Data Curation: High-quality data is a major bottleneck for multimodal foundation models, and open high-quality joint audio-visual supervision is rarely available.

3. Methodology

AV-Flamingo combines synchronized audio-visual encoding, temporal token interleaving, an LLM reasoner, and streaming speech synthesis. Its methodology also introduces long-form Audio-Visual-Skills data and timestamp-grounded TAVIT reasoning for complex audio-visual understanding.

  • Architecture: AV-Flamingo jointly processes visual and auditory inputs through five components, including SigLip, AF-Whisper, modality adaptors, an LLM, and streaming TTS.The architecture supports either or both modalities and voice-based interaction through speech synthesis.
  • Visual Encoding: Dynamic S2 encodes images and video at multiple spatial scales, then compresses the representations into compact tokens while preserving fine-grained spatial and temporal details.This enables higher-resolution images and longer videos with more frames without a proportional increase in LLM token consumption.
  • Audio Encoding: AF-Whisper standardizes audio as 16 kHz mono, converts it into 128-channel log-mel spectrograms, and uses sliding windows for arbitrary-length audio.The preprocessing uses a 25 ms window and 10 ms hop size.
  • Cross-Modal Fusion: Projected audio and visual tokens are divided into synchronized temporal chunks and interleaved by timestamp before entering Qwen2.5-7B as the LLM’s multimodal prefix.The modality-specific projections use 2-layer MLPs, while Qwen2.5-7B provides the core reasoning component.
  • Data and Reasoning: The methodology introduces AV-Skills-Short and AV-Skills-Long for reasoning-focused audio-visual QA, plus AV-Think for timestamp-grounded interleaved audio-visual reasoning.AV-Skills-Short contains videos up to 60 seconds, whereas AV-Skills-Long contains videos between 60 seconds and 15 minutes; AV-Think uses question-answer-reasoning triplets.

4. Experiments

The experiments train Audio-Visual Flamingo on 512 NVIDIA H100 GPUs and evaluate it against recent open-source, proprietary, and specialized audio-language baselines across diverse audio-visual, video, multimodal, and ASR benchmarks.

  • Experimental Setup: Training uses pre-training, long-context training, and post-training on 512 NVIDIA H100 GPUs.Batch sizes, learning rates, and optimizers are provided in Appendix C.
  • Baselines: The evaluation compares AV-Flamingo with recent open-source and proprietary multimodal language models, including Qwen2.5-O(mni), Qwen3-O(mni), OmniVinci, Gemini variants, and GPT-4o.Audio and speech comparisons additionally include Audio Flamingo series models and Qwen audio models.
  • Evaluation Datasets: Benchmarks cover audio understanding of sound, music, and speech reasoning through MMAR, MMSU, and MMAU.These datasets target audio reasoning capabilities.
  • Evaluation Datasets: Video benchmarks—Video-MME, ActivityNetQA, and LongVideoBench—assess visual comprehension and temporal reasoning.The benchmark set captures both general video understanding and long-range temporal reasoning.
  • Evaluation Datasets: WorldSense and DailyOmni evaluate joint reasoning across audio and video signals in omni/multimodal settings.These benchmarks test multimodal understanding beyond audio-only or video-only tasks.
  • Evaluation Datasets: ASR evaluation spans LibriSpeech, SPGISpeech, TEDLIUM, GigaSpeech, and Common Voice 15 across clean, noisy, and diverse real-world speech conditions.The datasets cover varied speech-recognition environments.

5. Results

AVF-Instruct transfers strongly across audio-only, video-only, and omni-modal tasks, achieving leading results on several benchmarks and maintaining competitiveness on long-form video understanding. Its results also indicate strong performance on long and complex real-world audio-visual settings.

  • Audio Understanding and Reasoning: AVF-Instruct improves over OmniVinci on MMAR (60.1 vs. 58.4) and exceeds Gemini 1.5 Pro on MMSU (61.5 vs. 60.7).
  • Audio Understanding and Reasoning: 73.49 is the best overall average score on MMAU, with especially strong gains in sound and speech understanding.AVF-Instruct outperforms both AF3 and OmniVinci on this measure.
  • Audio Understanding and Reasoning: AVF achieves the best ASR results on LibriSpeech test-clean (1.64 WER), SPGISpeech (2.8), and VoxPopuli (5.8).It remains close to the strongest baselines on other ASR evaluations.
  • Video and Omni Understanding: AVF-Instruct achieves the best Video-MME performance without subtitles (70.7) and with subtitles (71.2), outperforming NVILA and OmniVinci.On LongVideoBench, OmniVinci remains slightly stronger, while AVF-Instruct outperforms NVILA and remains competitive for long-form video understanding.
  • Video and Omni Understanding: Results on MMOU indicate that AVF is best-in-class for long and complex audio-visual understanding.The paper also reports qualitative examples and ablations of the compact training recipe and AV-Skills curriculum.

6. Conclusion, Limitations and Future Work

AV-Flamingo is presented as a fully open AV-LLM for joint understanding and reasoning over long, complex real-world videos, combining a large-scale dataset, staged curriculum, and timestamp-grounded reasoning. It achieves strong performance across diverse benchmarks, while remaining limited by data bias, difficult very-long-video reasoning, and incomplete real-world evaluation.

  • Contributions: AV-Flamingo combines AV-Skills, a three-stage curriculum, and TAVIT for timestamp-grounded reasoning over complex real-world videos.AV-Skills supports joint cross-modal learning, the curriculum scales from short-context perception to long-horizon reasoning, and TAVIT grounds intermediate thoughts to audio-visual timestamps.
  • Results: AV-Flamingo shows strong performance across diverse benchmarks, with especially notable gains on long and complex real-world videos.
  • Limitations: AV-Skills may introduce source bias and overlap with prior training data because it uses public datasets and open-internet videos.
  • Limitations: Reasoning over very long, highly dense videos remains challenging when evidence is sparse or temporally dispersed.
  • Future Work: Current benchmarks do not fully capture open-ended real-world deployment, motivating future expansion of AV-Skills to broader domains and more challenging tasks.

A. Audio-Visual Flamingo Training Datasets

The training datasets are organized by training stage and composition, with foundational data converted into standardized question-answer formats. Long-context supervised fine-tuning additionally uses AV-Safety QA to teach safe responses to harmful or privacy-sensitive audio-visual requests.

  • Dataset composition: Foundational datasets such as captioning and classification are converted into question-answer formats using the same prompts for each task.This standardizes heterogeneous training sources into a common QA formulation.
  • AV-Safety QA: 92K question–answer pairs over 536 hours of video comprise AV-Safety QA for long-context supervised fine-tuning.The subset probes unsafe, harmful, or policy-violating requests, including privacy-sensitive identification and sensitive-information extraction, with responses that safely decline or redirect.
  • Dataset composition: Table 2 summarizes the training datasets, including total hours, training-pair counts, and epochs used at each training stage.It covers both pre-training and fine-tuning datasets and their training composition.

B. Dataset Examples and Category Distribution

The section presents the training videos’ distribution across 15 major categories and gives representative examples from the AV-Skills-Long and AV-Skills-Short datasets.

  • Category Distribution: 15 major video categories are represented in the training datasets’ category distribution.Figure 4 visualizes the distribution of videos across these categories.
  • Dataset Examples: Representative examples are provided for both AV-Skills-Long and AV-Skills-Short.Figure 5 and Table 3 present these dataset examples.

C. Audio-Visual Flamingo Training Details

This section presents AV-Flamingo’s training settings across all stages, with stage-specific configurations and complementary tabular summaries.

  • Training settings: The section describes the model’s training settings across all stages.Each stage uses specific configurations.
  • Training settings: Table 4 provides detailed training configurations for the stages.
  • Training settings: Table 5 summarizes the training procedure as a compact stage-level recipe.

D. Additional Ablation Study

The ablation isolates the effects of curated AV-Skills-Short and AV-Skills-Long across the training curriculum. AV-Skills-Short improves audio-visual and video-only performance, while AV-Skills-Long further strengthens long-context reasoning and audio-visual grounding.

  • Ablation setup: The ablation compares OmniVinci, AVF-Stage1 after AV-Skills-Short pre-training, and AVF-Instruct after AV-Skills-Long mid-training.AVF-Stage1 combines Stages 1 and 2 with unimodal data, while AVF-Instruct is the final mid-trained model.
  • Ablation results: AV-Skills-Short yields consistent improvements over OmniVinci on audio-visual and video-only benchmarks.The result indicates that curated cross-modal data adds reasoning ability beyond unimodal supervision alone.
  • Ablation results: AV-Skills-Long further improves extended-temporal reasoning and tighter audio-visual grounding through eight long-form skill categories.The categories include Needle-in-the-Haystack, Temporal Order, Temporal Referring, Topic Summary, Counting, Sub-scene, Comparative, and Temporal Attribute.

E. GRPO-based RL training Details … H. Assets and Licenses

AV-Flamingo uses GRPO with group-normalized rewards to optimize structured reasoning, answer accuracy, and open-ended audio-visual generation. Its supporting sections specify timestamp-grounded multimodal data generation, TAVIT examples, prompt construction, and dataset licensing constraints.

  • E. GRPO-based RL training Details: AV-Flamingo trains with GRPO, estimating advantages from the average reward of multiple sampled outputs without an explicit value function.For each input, the policy generates a group of candidate outputs from the old policy.
  • E. GRPO-based RL training Details: 5 candidate outputs per input define the GRPO group, while within-group reward normalization stabilizes advantage computation.The clipping range, KL coefficient, and group size control optimization and policy regularization.
  • E. GRPO-based RL training Details: Format reward assigns 1 to outputs containing <think>...</think> reasoning followed by an <answer>...</answer> response, and 0 otherwise.The combined reward also uses normalized answer matching for QA and field-level JSON overlap for open-ended captions and responses.
  • E. GRPO-based RL training Details: The overall GRPO reward combines format with accuracy for QA and format with structured matching for open-ended generation, targeting correctness and grounded long-form quality.Structured matching averages field-level matches across categories after LLM-based JSON parsing.
  • F.1 AV-Think Generation Constraints: Long-form video generation combines timestamped speech, non-speech events, and visual captions into QA-reasoning triplets, multiple-choice variants, and integrated audio-visual captions.Generated questions must require both modalities and multi-hop temporal reasoning rather than surface-level transcription.
  • F.1 AV-Think Generation Constraints: Intermediate reasoning steps are grounded with <t>...</t> timestamp tags, while audio-only or vision-only questions are excluded.This constraint enforces temporal alignment and cross-modal dependence in AV-Think generation.
  • F.2 TAVIT Examples: TAVIT examples pair questions, grounded thinking chains, and ground-truth responses, with Table 8 adding five samples that link merged captions and source YouTube videos.The examples are truncated in the table, with full text available in the dataset.
  • G. Prompt Templates: AV-Skills-Long prompts curate synthetic QA pairs and answer options across categories, while Figure 20 specifies the prompt for AV-Think question–answer–reasoning triplets.Category-specific templates appear in Figures 6–19.

I. Broader Impacts

AV-Flamingo offers positive societal applications and broadens access to audio-visual intelligence through open release of its model, code, and dataset. Because joint audio-visual understanding also poses misuse risks, the authors restrict these resources to non-commercial research and prohibit harmful applications.

  • Positive societal applications: AV-Flamingo supports accessibility, education, and media analysis through applications including automated audio description, lecture and documentary understanding, captioning, and content moderation.These examples target visually impaired users, educational content, and automated media workflows.
  • Open access: Openly releasing the model, training code, and AV-Skills dataset lowers barriers for researchers without large proprietary resources and supports a more inclusive research ecosystem.The open release is framed as an access and inclusion benefit for research communities.
  • Risks and mitigation: Joint audio-visual understanding could enable surveillance, deepfake generation or verification, and multimodal disinformation at scale.These risks arise from jointly analyzing and synthesizing audio and visual content.
  • Risks and mitigation: AV-Flamingo and AV-Skills are released for non-commercial research use only under licenses explicitly prohibiting harmful applications.The licensing restriction is presented as a mitigation for the identified misuse risks.
Loading 2607.16107v1…