PaperlayerView exact revision on arXiv

Source-linked AI summary

Audio Flamingo 2: An Audio-Language Model with Long-Audio Understanding and Expert Reasoning Abilities

Sreyan Ghosh, Zhifeng Kong, Sonal Kumar, S Sakshi, Jaehyeon Kim, Wei Ping, Rafael Valle, Dinesh Manocha, Bryan Catanzaro

arXiv:2503.03983v1cs.SDcs.CLcs.LGeess.AS

TL;DR

Existing audio-language models and datasets provide limited support for understanding and reasoning over long audio. Audio Flamingo 2 combines an audio encoder, skill-specific question-answering data, and long-audio training data; it outperforms larger and proprietary models across more than 20 benchmarks.

  • Problem

    Prior audio-language models were limited to at most 30 seconds, and no existing work or datasets addressed long-audio understanding and reasoning.

    Source: §2 · p. 2

  • Method

    AF2 pairs a CLAP-based audio encoder with an audio-conditioned decoder language model and skill-specific AQA data, while LongAudio provides long-audio captioning and QA training instances.

    Source: §3 · p. 2, §1 · p. 2

  • Results

    AF2 outperforms larger and proprietary LALMs across over 20 benchmarks, and LongAudio-trained AF2 significantly outperforms its baseline on LongAudioBench.

    Source: §1 · p. 2

  • Takeaways & Limitations

    LongAudio and LongAudioBench establish dedicated training and evaluation datasets for advancing ALM understanding and reasoning over long audio.

    Source: §1 · p. 2

    The authors identify speech-content understanding, AudioSkills dataset diversity, and encoders inherently capable of processing long audio as areas for future improvement.

    Source: §7 · p. 9

Abstract

from arXiv · show

Understanding and reasoning over non-speech sounds and music are crucial for both humans and AI agents to interact effectively with their environments. In this paper, we introduce Audio Flamingo 2 (AF2), an Audio-Language Model (ALM) with advanced audio understanding and reasoning capabilities. AF2 leverages (i) a custom CLAP model, (ii) synthetic Audio QA data for fine-grained audio reasoning, and (iii) a multi-stage curriculum learning strategy. AF2 achieves state-of-the-art performance with only a 3B parameter small language model, surpassing large open-source and proprietary models across over 20 benchmarks. Next, for the first time, we extend audio understanding to long audio segments (30 secs to 5 mins) and propose LongAudio, a large and novel dataset for training ALMs on long audio captioning and question-answering tasks. Fine-tuning AF2 on LongAudio leads to exceptional performance on our proposed LongAudioBench, an expert annotated benchmark for evaluating ALMs on long audio understanding capabilities. We conduct extensive ablation studies to confirm the efficacy of our approach. Project Website: https://research.nvidia.com/labs/adlr/AF2/.

1. Introduction

Audio Flamingo 2 targets the shortage of high-quality training data and robust audio representations that limits expert-level reasoning over non-verbal audio. The paper also extends audio understanding to long clips with dedicated training and evaluation datasets.

1. Introduction

  • Expert-level audio reasoning remains difficult: even advanced LALMs underperform on reasoning relative to foundational event classification, amid weak data and audio representations.The introduction cites Gemini-1.5-Pro at 54.4% on MMAU sound and 48.5% on MMAU music.Source: §1 · pp. 1–2
  • AF2 combines a 3B-parameter decoder LM and 203M-parameter audio encoder with AudioSkills, AF-CLAP, and a three-stage curriculum.AudioSkills targets seven reasoning skills; AF-CLAP scales CLAP training to over 8M audio-caption pairs, uses synthetic data, and introduces an improved contrastive loss.Source: §1 · p. 2
  • LongAudio provides over 260k AQA instances from 30-second to 5-minute clips, while LongAudioBench supplies expert-annotated long-audio evaluation.LongAudio spans 10+ audio categories and supports six tasks: captioning and five reasoning-based QA tasks; AF2 trained on it significantly outperforms the baseline.Source: §1 · p. 2
  • AF2 outperforms larger and proprietary LALMs across over 20 benchmarks despite its smaller size and exclusive training on public datasets.The paper describes AF2 as a state-of-the-art audio-language model with advanced audio understanding and reasoning capabilities.Source: §1 · p. 2
  • The paper reports systematic ablation studies to assess the impact of each design choice.Source: §1 · p. 2

2. Related Work

Prior audio-language models have advanced foundational tasks, but expert-level reasoning and long-audio understanding remain gaps. Earlier systems generally use encoder-only shared audio-language spaces or decoder-based language models.

Audio-Language Models

  • Despite progress on classification and captioning, ALM advances have emphasized scaling model size and datasets, with less focus on data quality and audio representations, leaving expert reasoning weak.The passage identifies expert-level reasoning as important for advancing toward AGI.Source: §2 · p. 2
  • ALMs broadly divide into encoder-only models that align audio and language in a shared space, such as CLAP, and decoder-based models, pioneered by Pengi.Other cited examples include Wav2CLIP, AudioClip, and CoLLAT among encoder-only models, and LTU, SALMONN, AudioGPT, Audio Flamingo, Qwen-Audio, and Qwen-2-Audio among decoder-based models.Source: §2 · p. 2

Long Audio Understanding

  • Current ALMs generally process at most 30 seconds of audio, and most are limited to 10 seconds; the passage reports no existing work or datasets for long-audio understanding and reasoning.The limits are attributed to short-audio encoders and datasets dominated by clips of at most 10 seconds; long speech and long video have received attention.Source: §2 · p. 2

3. Audio Flamingo 2 Architecture

AF2 combines a CLAP-derived audio encoder, audio-feature transformation layers, a decoder-only language model, and gated cross-attention. Its design improves audio representations and conditions language-model processing on audio.

3. Audio Flamingo 2 Architecture

  • AF2 comprises AF-CLAP with sliding-window feature extraction, audio-representation transformation layers, a decoder-only language model, and gated XATTN-Dense layers for audio conditioning.Figure 2 summarizes the cross-attention architecture and three-stage curriculum training.Source: §3 · p. 2, Figure 2 · p. 3

3.1. AF-CLAP Audio Encoder

  • AF-CLAP addresses CLAP’s limited high-quality audio-caption data and resulting challenges in compositional reasoning and linguistic variation by scaling data and improving objective robustness.CLAP’s audio understanding and language alignment make it suitable for audio-language models, but the paper attributes its weaknesses to limited high-quality caption pairs.Source: §3.1 · p. 3

3.1.1. AF-CLAP TRAINING DATASET

  • AF-CLAP training uses over 8M 10-second audio-caption pairs, including approximately 5.5M new pairs: 4M from unlabeled short audios and 1.5M from long videos.The authors emphasize audio diversity and caption accuracy, addressing prior dataset limitations such as reliance on single-label classification captions and under-representation of home sounds.Source: §3.1.1 · p. 3

3.1.2. AF-CLAP TRAINING OBJECTIVE

  • AF-CLAP’s modified contrastive objective treats semantically equivalent caption variations as positives and captions with altered temporal or attribute composition as negatives.For each caption, the method generates M −1 same-semantics variations; for every M positive captions, it generates N composition-altered variations as negatives.Source: §3.1.2 · p. 3

3.2. Audio Conditioning Architecture & LLM

  • For each 10-second segment, AF2 extracts h ∈R64×2048 features, applies RoPE with base 4096, and uses up to 30 sliding windows for 5-minute LongAudio training.Three 8-head self-attention layers with inner dimension 2048 transform the audio representations.Source: §3.2 · p. 4
  • For 80 text tokens and 30×64 audio embeddings, XATTN-Dense uses about 1.5×10^5 attention complexity versus 4×10^6 for prefix tuning, reducing quadratic to linear complexity.Each gated cross-attention layer has residual cross-attention and dense blocks with tanh gating and is inserted before each LLM block.Source: §3.2 · p. 4
  • AF2 uses the frozen Qwen2.5-3B decoder-only causal language model, with 36 hidden layers and 16 attention heads.The paper describes it as a cost-performance choice with capacity for audio understanding while remaining lightweight.Source: §3.2 · p. 4

4. Audio Flamingo 2 Training Data

AF2 training data combines common audio benchmarks converted to AQA format with two proposed datasets: AudioSkills for expert reasoning and LongAudio for long-audio understanding. Together, these datasets target skill-specific reasoning on short clips and captioning and question answering on long audio.

4. Audio Flamingo 2 Training Data

  • AF2 training also converts common Audio Flamingo benchmark datasets into AQA format, alongside the new AudioSkills and LongAudio datasets.AudioSkills targets audio expert reasoning, while LongAudio targets long-audio understanding.Source: §4 · p. 4

4.1. AudioSkills: An Expert Audio Reasoning Dataset

  • AudioSkills contains ≈4.2M synthetic QA pairs for seven reasoning skills, built from open-source sound and music datasets and synthetic audio for clips ≤30 seconds.The dataset is designed to develop reasoning and problem-solving abilities beyond surface-level acoustic properties.Source: §4.1 · pp. 4–5

4.2. LongAudio: A Long Audio Understanding Dataset

  • LongAudio contains over 80K unique audios and approximately 263K AQA pairs sourced from long-video datasets, with questions targeting captioning, reasoning, and information extraction.The source videos include a subset of MiraData and all of Video ReCap; Qwen2-VL-2B-Instruct and Qwen2-Audio generate segment captions, which GPT-4o uses to create questions.Source: §4.2 · p. 5
  • LongAudioBench’s final version has 2,429 expert-verified instances, evaluated by GPT-4o as a judge on a 1-to-10 scale.Annotators checked and corrected a proportional 10% sample (≈3K) of LongAudio QA pairs, discarding incorrect items before author quality checks.Source: §4.2 · p. 6

5. Audio Flamingo 2 Training Strategy

AF2 uses a three-stage curriculum that progressively increases audio context length while improving data quality, moving from multimodal alignment to short-audio reasoning and then long-audio reasoning.

Stage 1: Pre-training

  • Stage 1 aligns audio representations with the LLM using large-scale classification and captioning data plus a small amount of high-quality QA data.CLAP and LLM layers are frozen; only audio representation transformation and gated cross-attention layers are trainable, with context limited to 3 windows (30 seconds).Source: §5 · p. 6

Stage 2: Fine-tuning

  • Stage 2 develops audio understanding and reasoning skills through high-quality short-audio classification, captioning, and QA datasets.Only the LLM is frozen; CLAP, audio representation transformation, and gated cross-attention layers are trainable, with context increased to 9 windows (1.5 minutes).Source: §5 · pp. 6–7

Stage 3: Long Fine-tuning

  • Stage 3 extends context and teaches long-audio reasoning skills using the proposed LongAudio dataset.Audio representation transformation and gated cross-attention layers remain trainable, and context increases to 30 windows (5 minutes).Source: §5 · p. 7

6. Experiments

AF2 outperforms other LALMs on audio reasoning benchmarks despite its smaller LLM, while experiments examine the contributions of audio representations, training data, conditioning, model scale, and training schedules. Long-audio evaluation uses a cascaded approach for models that cannot process long inputs directly.

6.1. Experimental Setup

  • AF2 is trained on 128 A100 GPUs and evaluated against recent LALMs, with zero-shot datasets excluded from training.For LongAudioBench, models unable to process inputs of at least 30 seconds use captions of short segments to answer questions; the paper reports that passing original long audio consistently performed better. Gemini receives the full audio. Fine-tuning uses dynamic batching by audio length.Source: §6.1 · p. 7

6.2. Smaller but Better

  • With a much smaller LLM, AF2 outperforms all evaluated LALMs by large margins on audio reasoning benchmarks.On foundational audio-understanding benchmarks, AF2 is competitive with state-of-the-art LALMs despite having half their size. Evaluations span foundational audio-understanding and reasoning datasets, including LongAudioBench.Source: Table 2 · p. 7, §6.2 · p. 7

6.3. Enhanced Audio Features Boost Performance

  • Replacing AF-CLAP with other CLAP models reduces performance across benchmarks, underscoring the importance of robust audio representations.The comparison includes AF-CLAP trained with 630k audio-text pairs and a version using the standard contrastive loss.Source: §6.3 · pp. 7–8, Table 3 · p. 8

6.4. High Quality Data Boosts Reasoning Abilities

  • AudioSkills improves reasoning, data diversity benefits challenging reasoning and zero-shot classification, and more training data consistently improves performance.Using half the data outperforms using only OpenAQA on challenging tasks.Source: Table 4 · p. 8, §6.4 · p. 8

6.5. Cross-Attention Outperforms Prefix-Tuning

  • Trained on the same data and recipe, AF2 significantly outperforms 7B GAMA, which uses prefix tuning.The paper attributes AF2’s advantage to superior audio representations and shift cross-attention-based conditioning; GAMA’s setup excludes stage 3.Source: Table 5 · p. 8, §6.5 · p. 8

6.6. Effect of Scaling LLM

  • AudioSkills improves overall and reasoning performance across LLM sizes, while reasoning without it depends more heavily on model size.The comparison covers AF2 models from 0.5B to 7B parameters.Source: Figure 7 · p. 8, §6.6 · p. 8

6.7. Effect of Training Schedules

  • Training-schedule experiments find that LLM fine-tuning helps classification and captioning but hurts reasoning, pre-training alignment is essential, and gradual context extension supports long-audio understanding.The study compares 10 schedules, including a four-stage curriculum that repeats stage 2 with LLM fine-tuning and shifts long fine-tuning to stage 4.Source: Table 6 · p. 8, §6.7 · p. 8

7. Conclusion, Limitations and Future Work … B.4. Comparison with prior-art

Audio Flamingo 2 combines a custom CLAP and synthetic reasoning data for long-audio understanding and expert reasoning, while LongAudio and LongAudioBench target reasoning over long contexts. The paper reports strong AF-CLAP results and identifies speech understanding, AudioSkills diversity, and long-audio encoders as future-work priorities.

7. Conclusion, Limitations and Future Work

  • Audio Flamingo 2 uses a custom CLAP trained on a large-scale dataset with a novel objective and synthetic reasoning AQA data, and the paper proposes LongAudio and LongAudioBench for long-audio reasoning.The paper characterizes Audio Flamingo 2 as achieving state-of-the-art audio understanding and reasoning despite its small model footprint.Source: §7 · pp. 8–9
  • Future work targets better speech-content understanding, more diverse AudioSkills datasets, and audio encoders inherently capable of processing long audio.Source: §7 · p. 9

B.1. Training Hyper-parameters

  • AF-CLAP was trained for 12 epochs on 8 A100 80GB GPUs, with a 5e-4 learning rate, M=N=3, and effective batch size 256.The batch size was smaller than in prior literature because of computational constraints.Source: §B.1 · p. 15

B.2. Feature Extraction Layer

  • Using dense features from the final audio-encoder layer significantly improves AF2 performance over using CLAP-head features.AF2 discards the CLAP head and uses dense features from the final HTS-AT audio-encoder layer.Source: §B.2 · p. 15

B.3. Training Datasets

  • Table 9 provides detailed statistics for the datasets used to train AF-CLAP.The supplied passage does not provide the dataset statistics themselves.Source: §B.3 · p. 15

B.4. Comparison with prior-art

  • AF-CLAP achieves state-of-the-art performance across all reported audio-to-text and text-to-audio retrieval metrics on Clotho and AudioCaps.The 630k-pair variant does not improve over Laion-CLAP; the authors attribute this to their data augmentation and cleaning applying to only a small portion of Laion-CLAP’s captions.Source: §B.4 · p. 15
  • AF-CLAP consistently achieves state-of-the-art results on zero-shot audio classification benchmarks.The authors make the same conclusion about Laion-CLAP as for the retrieval comparison.Source: §B.4 · p. 15
  • Benchmark datasets do not holistically evaluate CLAP capabilities, while AF-CLAP features are described as more robust and as enhancing (L)ALM audio perception.The passage cites Ghosh et al. (2024d), Wu et al. (2023a), Selvakumar et al. (2024), and Ghosh et al. (2025) on benchmark limitations.Source: §B.4 · p. 15

C. Limitations of CLAP Retrieval Based Evaluation · D. AudioSkills

CLAP-based retrieval can mis-score AF2’s open-ended responses when it fails to match a correct prediction to the right label. Fine-tuning can also encourage style memorization that favors the retrieval-based evaluation.

C. Limitations of CLAP Retrieval Based Evaluation

D.1. Dataset Statistics … E.3. Comparison with other datasets

The appendix documents dataset statistics, generation metadata, and examples for AudioSkills and LongAudio, and reports that LongAudio has the longest average audio duration among compared AQA datasets.

D.1. Dataset Statistics

  • Table 14 presents category-wise AudioSkills statistics and metadata used for data generation; transcripts come from Whisper Large-v3, while audio and visual captions come from Qwen2-Audio and Qwen2-VL.Source: §D.1 · pp. 15–16
  • Table 7 compares AF-CLAP baselines on Text-to-Audio and Audio-to-Text retrieval using AudioCaps and Clotho.Source: Table 7 · p. 16
  • Table 8 compares AF2 CLAP feature extraction methods, distinguishing head features from dense features taken from the audio encoder’s last layer.Source: Table 8 · p. 16
  • Table 9 reports statistics for audio-caption datasets used to train CLAP, noting that captioned datasets underwent cleaning and marking collected datasets with an asterisk.Source: Table 9 · p. 16

E.1. Detailed Statistics

  • Table 11 presents LongAudio and LongAudioBench statistics categorized by QA type.Source: §E.1 · p. 16

E.2. Examples

  • Table 15 provides category-wise examples from LongAudio.Source: §E.2 · p. 16

E.3. Comparison with other datasets

  • LongAudio has the longest average audio duration among the AQA datasets compared in Table 12.The passage provides no duration values for the comparison.Source: §E.3 · p. 16

E.4. Fine-grained results for AF2 … G.3. Effect of Cross-Attention Frequency

LongAudio fine-tuning improves AF2 across LongAudioBench categories, especially on novel QA tasks. The appendix also documents dataset preparation, expert verification, and model-design ablations.

E.5. Success and Failure Cases on LongAudioBench

  • Table 22 presents success and failure cases of AF2 on LongAudioBench.Source: §E.5 · p. 16

E.6. Metadata

  • MiraData metadata uses visual captions, while Video ReCap metadata uses action captions; the appendix also reports dataset and duration statistics, with LongAudio having the longest average duration.It also includes a CLAP comparison on zero-shot audio-classification benchmarks.Source: §E.6 · p. 16, Table 12 · p. 17, Table 11 · p. 17

E.4. Fine-grained results for AF2

  • LongAudio fine-tuning improves AF2 scores across all LongAudioBench categories, particularly on the novel NeedleQA and SubsceneQA tasks.These tasks were not encountered during AF2’s two-stage training.Source: §E.4 · p. 16, Table 13 · p. 17

E.7. Human Verification for LongAudioBench

  • LongAudioBench QA pairs underwent two expert review stages, with each pair seen by at least two people and checked for audio validity, ambiguity, format, and quality.Review guidelines also required listening to the complete audio and escalating ambiguous or complex pairs for collective review.Source: §E.7 · pp. 17–18

F.1. Prompts for foundational datasets; F.2. Prompts for other tasks

  • Prompts convert foundational audio datasets into AQA and support LongAudio QA generation, AudioSkills reasoning, caption cleaning, and CLAP training examples.The listed prompts cover tasks including temporal QA, contextual sound-event reasoning, complex QA, and generating caption negatives.Source: §F.1 · p. 18, §F.2 · pp. 18–19

G.1. Effect of RoPE

  • Without RoPE in the audio transformation layers, AF2’s performance drops on LongAudioBench and MMAU.The authors identify RoPE as important for long-context and reasoning capabilities.Source: Table 16 · p. 20, §G.1 · p. 20

G.2. Effect of Audio Transformation Layers

  • Removing the audio transformation layers reduces AF2’s performance, highlighting their role in audio representation learning and adaptation.The comparison is reported in Table 17.Source: Table 17 · p. 20, §G.2 · p. 20

G.3. Effect of Cross-Attention Frequency

  • Cross-attention after every language-model layer performs best; every third layer is competitive, while every sixth produces a noticeable drop.The compared conditioning frequencies are reported in Table 18.Source: Table 18 · p. 20, §G.3 · p. 20

G.4. Results on different LLM Sizes · H. More Details · H.1. Topic-wise Video Segmentation

The section compares AF2 across LLM sizes from 0.5B to 7B and describes a caption-based, multi-step process for segmenting videos into topic-specific clusters.

G.4. Results on different LLM Sizes

  • Table 19 compares AF2 performance across LLM sizes ranging from 0.5B to 7B.The supplied passage identifies the comparison's scope but does not provide the table's results.Source: Table 19 · p. 20, §G.4 · p. 20

H.1. Topic-wise Video Segmentation

  • Topic-specific video clusters were built through a multi-step process using textual video captions.The steps include caption embedding, K-Means clustering, representative video selection, and manual review.Source: §H.1 · p. 20
  • Video captions were represented with semantic feature embeddings extracted using NV-Embed (Lee et al., 2024).Source: §H.1 · p. 20
  • K-Means clustering grouped videos according to their feature representations.Source: §H.1 · p. 20
  • Representative subsets combined random sampling and DPP diversity scoring (Kulesza et al., 2012) to capture cluster topics while avoiding redundancy.Clusters were analyzed for relevance and diversity before selection.Source: §H.1 · p. 20
  • A random sample from each cluster was manually reviewed, with ambiguous or low-quality videos discarded or reassigned.Source: §H.1 · p. 20

H.2. Data Loader · H.3. Computational Analysis

The appendix describes a weighted, length-aware loader with dynamic batch constraints, reporting a reduction in padding; it also gives staged training times and qualifies its cost estimate.

H.2. Data Loader

  • AF2 training audio spans 0.5 seconds to 10 minutes; a generic loader would cause excessive padding and unstable losses, so batches are capped by job duration.The loader groups similar-length samples to minimize padding.Source: §H.2 · p. 21
  • Each dataset receives a per-epoch sampling weight, and its duration buckets contribute controlled slices indexed by epoch and bucket size.Clips are bucketed by duration to reduce padding overhead.Source: §H.2 · p. 21
  • When a bucket cycles, it is shuffled deterministically using a seed from the dataset name and current epoch, providing reproducibility with randomness across epochs.Source: §H.2 · p. 21
  • Batches group examples by token or frame length, subject to maximum example count, maximum token budget, and optional batch-size alignment for multi-GPU efficiency.A batch is finalized when adding an example would exceed a count or token limit; remaining examples form the final batch.Source: §H.2 · p. 21
  • Padding fell from 58% to 16% per batch with the dynamic batching scheme.This is the reported effectiveness of the loader.Source: §H.2 · p. 21

H.3. Computational Analysis

  • For the 3B model, stage 1 took about 4 days, stage 2 about 5 days, and stage 3 about 12 hours on 128 NVIDIA A100 80GB GPUs.Estimated experimentation cost is about 5× final model training cost, excluding language-model training costs and other failures.Source: §H.3 · p. 21, Table 20 · p. 21

Related papers

Loading 2503.03983v1…