Source-linked AI summary
Audio Flamingo 2: An Audio-Language Model with Long-Audio Understanding and Expert Reasoning Abilities
Sreyan Ghosh, Zhifeng Kong, Sonal Kumar, S Sakshi, Jaehyeon Kim, Wei Ping, Rafael Valle, Dinesh Manocha, Bryan Catanzaro
TL;DR
Audio-language models have been limited in their ability to reason over complex and long audio. AF2 combines specialized audio representations, reasoning-oriented data, and curriculum training, achieving state-of-the-art performance with a small language model and extending audio understanding to five-minute segments.
Problem
Existing audio-language models have limited support for long audio, while high-quality data and robust representations for complex audio reasoning remain insufficient.
Method
AF2 combines AF-CLAP, synthetic skill-specific Audio QA data, and a three-stage curriculum, and introduces LongAudio for long-audio training.
Results
AF2 achieves state-of-the-art audio understanding and reasoning with a 3B-parameter language model and outperforms larger and proprietary models across over 20 benchmarks.
Takeaways & Limitations
LongAudio and LongAudioBench extend audio understanding and evaluation to segments lasting from 30 seconds to 5 minutes.
Takeaways & Limitations
The authors identify speech understanding, AudioSkills diversity, and inherently long-audio-capable encoders as areas for future work.
Abstract
from arXiv · showhide
Understanding and reasoning over non-speech sounds and music are crucial for both humans and AI agents to interact effectively with their environments. In this paper, we introduce Audio Flamingo 2 (AF2), an Audio-Language Model (ALM) with advanced audio understanding and reasoning capabilities. AF2 leverages (i) a custom CLAP model, (ii) synthetic Audio QA data for fine-grained audio reasoning, and (iii) a multi-stage curriculum learning strategy. AF2 achieves state-of-the-art performance with only a 3B parameter small language model, surpassing large open-source and proprietary models across over 20 benchmarks. Next, for the first time, we extend audio understanding to long audio segments (30 secs to 5 mins) and propose LongAudio, a large and novel dataset for training ALMs on long audio captioning and question-answering tasks. Fine-tuning AF2 on LongAudio leads to exceptional performance on our proposed LongAudioBench, an expert annotated benchmark for evaluating ALMs on long audio understanding capabilities. We conduct extensive ablation studies to confirm the efficacy of our approach. Project Website: https://research.nvidia.com/labs/adlr/AF2/.
1. Introduction
Audio Flamingo 2 targets the limited ability of existing systems to understand and reason over non-speech audio and music. It combines specialized data, audio representation learning, and curriculum training, while extending evaluation to long audio.
- Audio understanding covers non-speech sounds, non-verbal speech, and music, which support applications including anomaly detection, emotional-cue recognition, and assistive technologies.
- AF2 combines a 3B-parameter decoder language model with a 203M-parameter audio encoder for parameter-efficient audio understanding and reasoning.
- AudioSkills provides skill-specific AQA data with complex reasoning questions targeting seven distinct audio reasoning skills.
- AF-CLAP scales CLAP training to over 8M audio-caption pairs, uses synthetic data, and applies an improved contrastive loss for robust representations.
- A three-stage curriculum progressively improves multimodal alignment, data quality, audio context length, and long-audio reasoning.
- AF2 extends audio understanding from short clips to audio lasting up to 5 minutes through LongAudio and LongAudioBench.
- AF2 outperforms larger and proprietary language-audio models across over 20 benchmarks despite using a smaller model and only public training datasets.
2. Related Work
Prior audio-language models generally focus on short audio, while long-audio understanding and reasoning lack dedicated datasets and benchmarks. Existing CLAP-based approaches also face limitations in complex audio representation and compositional reasoning.
- Audio-language models include encoder-only systems such as CLAP, which learn shared audio-language representations for retrieval and zero-shot classification.
- Most current ALMs process at most 30 seconds of audio, with many limited to 10 seconds, and no prior work or datasets addressed long-audio understanding and reasoning.
3. Audio Flamingo 2 Architecture
AF2 combines a CLAP-based encoder, representation transformations, a decoder language model, and gated cross-attention for audio conditioning. AF-CLAP improves training data and contrastive objectives to support linguistic invariance and compositional reasoning.
- AF2 uses AF-CLAP with sliding-window extraction, transformation layers, a decoder-only language model, and gated cross-attention for audio conditioning.
- AF-CLAP is trained on over 8M 10-second audio-caption pairs collected from open audio and video datasets.
- AF-CLAP training objective: AF-CLAP treats linguistically varied captions with identical semantics as positives to improve linguistic invariance.
- AF-CLAP training objective: Composition-aware negatives modify temporal or attribute relationships between acoustic events to improve compositional reasoning.
- The training pipeline segments long and short videos into 10-second clips, generates audio-centric captions, and trains CLAP with a modified contrastive loss.
- For longer audio, AF2 concatenates features from non-overlapping sliding windows, reaching up to 30 windows or 5 minutes during LongAudio training.
- Gated cross-attention reduces the stated prefix-tuning complexity from 4×10^6 to approximately 1.5×10^5 for the given text and audio lengths.
- The decoder language model is Qwen2.5-3B, selected for its stated cost-performance trade-off and used with 3B parameters.
4. Audio Flamingo 2 Training Data
Audio Flamingo 2 uses AudioSkills and LongAudio to train fine-grained reasoning over short and long audio. These datasets target expert skills, captioning, and multiple forms of long-audio question answering.
- AudioSkills: Existing audio datasets and derived QA pairs often emphasize surface properties rather than expert-level reasoning.
- AudioSkills: AudioSkills is a high-quality synthetic dataset designed to develop audio reasoning and problem-solving skills across seven skill categories.It focuses on audios ≤30 seconds and uses open-source sound and music datasets, synthetic audio, metadata, and GPT-4o-generated QA pairs.
- AudioSkills: AudioSkills contains approximately 4.2M QA pairs whose questions are designed to require deliberate reasoning.
- LongAudio: LongAudio is generated by segmenting long videos into approximately 10-second clips, captioning them, and using an LLM to generate QA pairs.A sampled subset receives expert verification and quality checks to form LongAudioBench, whose final version has 2,429 instances.
5. Audio Flamingo 2 Training Strategy
AF2 is trained with a curriculum that progressively increases audio context length while improving data quality. Its long fine-tuning stage uses LongAudio to teach reasoning over five-minute audios.
- Curriculum learning: AF2 uses a three-stage curriculum learning strategy that progressively increases audio context length and improves data quality.
- Evaluation: Table 1 evaluates foundational audio understanding using captioning, audio question answering, classification, and zero-shot metrics.
- Long fine-tuning: The long fine-tuning stage trains on LongAudio while increasing the audio context to w = 30 windows, or 5 minutes.
6. Experiments
Experiments evaluate AF2 on foundational understanding, expert reasoning, audio representations, data quality, model size, and training schedules. AF2 is competitive or superior across these comparisons despite its smaller language model.
- Evaluation: AF2 is evaluated against recent LALMs using foundational and reasoning benchmarks spanning sound, music, speech, captioning, classification, and audio QA.The evaluation includes more than 20 benchmark settings, including LongAudioBench.
- Foundational audio understanding: AF2 achieves competitive foundational audio understanding while using half the size of several compared LALMs.
- Expert reasoning: AF2 outperforms all compared LALMs by large margins on audio reasoning benchmarks despite using a much smaller LLM.
- Audio representations: Replacing AF-CLAP with other CLAP models causes performance drops across benchmarks, supporting the importance of robust audio representations.
- Model comparison: AF2 significantly outperforms GAMA trained with the same data and recipe, which the authors attribute to superior audio representations and cross-attention conditioning.
- Data quality and model size: AudioSkills improves overall and reasoning performance across LLM sizes, while excluding it makes reasoning performance depend more strongly on model size.
7. Conclusion, Limitations and Future Work
Audio Flamingo 2 combines a custom CLAP, synthetic reasoning data, and long-audio datasets to support audio understanding and expert reasoning. The authors identify speech understanding, dataset diversity, and long-audio encoder design as future limitations to address.
- Conclusion: AF2 combines a custom CLAP trained with a novel objective, synthetic reasoning AQA data, and dedicated datasets for long-audio understanding.
- Conclusion: AF2 achieves state-of-the-art audio understanding and reasoning performance despite its small model footprint.
- Limitations and future work: Future work includes improving speech content understanding, expanding AudioSkills to more diverse datasets, and developing audio encoders that inherently process long audio.
B.1. Training Hyper-parameters
AF-CLAP is trained under computational constraints, and dense encoder features outperform CLAP-head features while improving audio-language performance across retrieval and classification benchmarks.
- Training setup: AF-CLAP is trained on 8 A100 80GB GPUs for 12 epochs with a 5e-4 learning rate and effective batch size of 256.The batch size is smaller than in prior literature because of computational constraints.
- Audio features: Dense features from the final audio-encoder layer significantly improve AF2 performance compared with CLAP-head features.AF2 discards the CLAP head and uses dense HTS-AT features instead.
- Benchmark results: AF-CLAP achieves state-of-the-art results across all reported AudioCaps and Clotho retrieval metrics.The comparison includes standard contrastive training, noise-reduction ablation, and a 630k-pair variant.
- Benchmark results: AF-CLAP consistently achieves state-of-the-art results on zero-shot audio-classification benchmarks.The authors note that the same conclusion applies to Laion-CLAP in this comparison.
- Interpretation: AF-CLAP’s robust audio features enhance audio perception in language and audio-language models beyond benchmark-specific evaluation.The authors caution that benchmark datasets do not holistically evaluate CLAP capabilities.
C. Limitations of CLAP Retrieval Based Evaluation
AF2’s open-ended outputs are evaluated by CLAP-based label retrieval, but this procedure can misclassify correct answers and can favor fine-tuned output styles.
- Evaluation procedure: CLAP-based evaluation retrieves a label by comparing AF2’s open-ended response with candidate labels and selecting the most similar one.This evaluation scheme is widely adopted for prior audio-language-model benchmarks.
- Failure cases: Correct AF2 predictions can receive incorrect labels when semantic similarity retrieval fails to match the intended class.Examples include Rock becoming Metal on GTZAN and Pouring water becoming Water drops on ESC50.
- Evaluation bias: Fine-tuning the language model causes style memorization that favors the CLAP retrieval evaluation method.The paper shows prediction shifts toward label-like outputs that increase measured accuracy.
D.1. Dataset Statistics
The appendix documents datasets and feature variants used in AF-CLAP and LongAudio, whose defining dataset-level distinction is substantially longer audio duration.
- Feature variants: Table 8 compares AF2 using CLAP-head features with dense features from the audio encoder’s last layer.Dense features are the feature-extraction variant associated with improved AF2 performance.
- AF-CLAP datasets: AF-CLAP uses audio-caption datasets whose statistics are documented in Table 9, with caption cleaning applied across the listed datasets.An asterisk marks datasets collected by the authors.
- Long-audio datasets: LongAudio and LongAudioBench statistics are categorized by question-answer types and benchmark categories.The appendix also provides category-wise examples from LongAudio.
- Duration statistics: LongAudio has the longest average audio durations among the compared audio question-answering datasets.The comparison is presented in Table 12.
E.4. Fine-grained results for AF2
AF2’s LongAudioBench results improve after LongAudio fine-tuning, especially on tasks absent from its earlier training, while the appendix details dataset construction and architectural ablations.
- Fine-grained results: Fine-tuning AF2 on LongAudio improves LongAudioBench scores across all categories, especially NeedleQA and SubsceneQA.These tasks were not encountered during AF2’s two-stage training.
- Fine-grained results: The LongAudioBench analysis includes category-wise results and success and failure cases for AF2.These materials provide both aggregate category comparisons and qualitative examples.
- Benchmark construction: LongAudioBench uses expert annotation with correction, verification, and review stages, with each QA pair annotated by at least two individuals.An annotation tool supports the process, which follows explicit accuracy, listening, validation, and answer-format guidelines.
- Ablations: Removing RoPE from the audio transformation layers reduces performance on LongAudioBench and MMAU.The ablation highlights RoPE’s importance for long-context and reasoning capabilities.
- Ablations: Removing the audio transformation layers reduces performance, indicating their role in expanding audio representation learning and adaptation capacity.The comparison is reported in Table 17.
- Ablations: Conditioning audio representations after every language-model layer performs best, while every third layer remains competitive and every sixth layer shows a noticeable drop.This comparison evaluates cross-attention conditioning frequencies.
G.4. Results on different LLM Sizes
This section compares AF2 performance across language-model sizes from 0.5B to 7B and describes a clustering procedure for selecting diverse videos.
- Results on different LLM Sizes: AF2 performance is evaluated across language-model sizes ranging from 0.5B to 7B.The comparison is reported in Table 19.
- Video Selection: Video captions are embedded with NV-Embed to obtain semantic feature representations.These representations are used as inputs to the subsequent clustering step.
- Video Selection: K-Means groups videos according to their caption-derived feature representations.The clustering step organizes videos based on semantic similarity.
- Video Selection: Representative videos are selected from each cluster using random sampling and Determinantal Point Processes to preserve diversity and reduce redundancy.The selected subsets are intended to capture the range of topics in each cluster.
- Video Selection: Manual review removes or reassigns videos with ambiguous or low-quality content.A random sample from each cluster is reviewed for quality and relevance.
H.2. Data Loader
The data loader blends weighted, duration-bucketed datasets and dynamically batches similarly sized audio examples. This design controls sampling, improves shuffling, reduces padding, and supports audio ranging from 0.5 seconds to 10 minutes.
- Data Loader: AF2 training uses audio durations from 0.5 seconds to 10 minutes, making generic batching prone to excessive padding and unstable losses.The loader addresses this with dynamic batching and job-specific maximum-duration constraints.
- Weighted Bucketed Blending: Duration buckets group similarly sized clips, while dataset weights control how many examples each dataset contributes per epoch.This weighted bucketed blending reduces padding overhead before mini-batches are formed.
- Weighted Bucketed Blending: Bucket slices are shuffled deterministically when a complete bucket cycle is reached, preserving randomness across epochs and reproducibility.The shuffle seed is derived from the dataset name and current epoch.
- Dynamic Batching: Dynamic batching incrementally groups examples under maximum-token, maximum-sentence, and optional batch-size-multiple constraints.The batch-size multiplier supports alignment requirements, including multi-GPU efficiency.
- Effectiveness: 58% to 16%: dynamic batching reduces the percentage of padding in each batch.The loader combines weighted bucketed blending with length-aware batching to improve training efficiency.