Source-linked AI summary
Audio Flamingo 3: Advancing Audio Intelligence with Fully Open Large Audio Language Models
Arushi Goel, Sreyan Ghosh, Jaehyeon Kim, Sonal Kumar, Zhifeng Kong, Sang-gil Lee, Chao-Han Huck Yang, Ramani Duraiswami, Dinesh Manocha, Rafael Valle, Bryan Catanzaro
TL;DR
Audio Flamingo 3 addresses gaps in audio-language models’ reasoning, long-context, multimodal, interaction, and openness capabilities. It combines a unified AF-Whisper encoder, curated reasoning and dialogue datasets, and five-stage curriculum training. AF3 achieves state-of-the-art results across 20+ benchmarks while remaining fully open, though its release is limited to research use and future work identifies remaining system and language constraints.
Problem
Existing LALMs have limited expert reasoning, multi-turn multi-audio dialogue, long-audio understanding, unified modality coverage, and transparent access to data and training recipes.
Method
AF3 combines unified AF-Whisper representations with AudioSkills-XL, LongAudio-XL, AF-Think, AF-Chat, and a five-stage curriculum.
Results
AF3 achieves new state-of-the-art results across 20+ audio understanding and reasoning benchmarks and outperforms most open-weight and closed-source LALMs while being fully open.
Takeaways & Limitations
AF3 provides a fully open LALM spanning speech, sounds, and music with long-audio reasoning up to 10 minutes, multi-audio chat, on-demand thinking, and voice interaction.
Takeaways & Limitations
AF3 releases are under a research-only license because of the licensing and scope of its training data.
Abstract
from arXiv · showhide
We present Audio Flamingo 3 (AF3), a fully open state-of-the-art (SOTA) large audio-language model that advances reasoning and understanding across speech, sound, and music. AF3 introduces: (i) AF-Whisper, a unified audio encoder trained using a novel strategy for joint representation learning across all 3 modalities of speech, sound, and music; (ii) flexible, on-demand thinking, allowing the model to do chain-of-thought-type reasoning before answering; (iii) multi-turn, multi-audio chat; (iv) long audio understanding and reasoning (including speech) up to 10 minutes; and (v) voice-to-voice interaction. To enable these capabilities, we propose several large-scale training datasets curated using novel strategies, including AudioSkills-XL, LongAudio-XL, AF-Think, and AF-Chat, and train AF3 with a novel five-stage curriculum-based training strategy. Trained on only open-source audio data, AF3 achieves new SOTA results on over 20+ (long) audio understanding and reasoning benchmarks, surpassing both open-weight and closed-source models trained on much larger datasets.
1 Introduction
Audio Flamingo 3 addresses limitations in audio-language models by combining broad audio reasoning with openness and new interaction capabilities. It targets speech, sound, and music across diverse benchmarks and contexts.
- Motivation and Contributions: Existing LALMs remain limited in expert reasoning, multi-turn multi-audio dialogue, and long-audio understanding, while many lack transparent data and training recipes.The paper attributes these gaps partly to short, recognition-focused training and insufficient exposure to complex audio skills.
- Motivation and Contributions: AF3 is a fully open-source large audio-language model designed for audio understanding and reasoning across speech, sounds, and music.Its release commitment covers model weights, training data, code, and training methodology, under a research-only license.
- Motivation and Contributions: AF3 introduces AudioSkills-XL, LongAudio-XL, AF-Think, and AF-Chat alongside a unified AF-Whisper encoder and a five-stage curriculum.The datasets target diverse audio QA, long-audio reasoning, controlled thinking prefixes, and multi-turn multi-audio chat.
- Motivation and Contributions: AF3 supports on-demand thinking, multi-turn multi-audio chat, long-context audio reasoning up to 10 minutes, and voice-to-voice interaction.These capabilities are presented as extensions beyond standard short-audio question answering.
2 Related Work
Related work establishes audio-language modeling as a route to reasoning over speech, sounds, and music, but current systems remain constrained in capability coverage and openness. AF3 combines long-context training, modality diversity, and controlled reasoning supervision to address these gaps.
- Audio Language Models: Audio-language models extend language models to auditory inputs, with encoder-only systems supporting shared audio-text embeddings and retrieval.CLAP, Wav2CLIP, and AudioCLIP represent the encoder-only paradigm.
- Audio Language Models: Current LALMs often lack multi-turn multi-audio chat, long-context comprehension, unified speech-sound-music understanding, and transparent training resources.Partially open releases may provide checkpoints without accompanying data or code, limiting reproducibility.
- Reasoning and Long-Context Understanding: Prior long-context audio work advanced comprehension but remained limited to sounds and music rather than speech.AF2 is identified as an early step toward long-context audio understanding.
- Reasoning and Long-Context Understanding: AF3 integrates controlled reasoning supervision, long-context training, and modality diversity to strengthen audio reasoning and long-context comprehension.The approach builds on reasoning datasets, multimodal perception, and chain-of-thought-style prompting.
3 Methodology
AF3 combines a unified audio encoder, an audio-to-text adaptor, a decoder-only language model, and streaming text-to-speech. Its architecture processes diverse audio through AF-Whisper before language generation and voice output.
- Architecture: AF3 consists of AF-Whisper with sliding-window extraction, audio projector layers, a large language model, and streaming TTS.The architecture is summarized in Figure 2.
- AF-Whisper Audio Encoder: AF-Whisper unifies speech, sounds, and music in one encoder to reduce complexity, framerate mismatches, and training instability associated with separate encoders.It is initialized from Whisper large-v3 and trained through audio captioning with next-token prediction.
- Feature Extraction: Audio inputs are resampled to 16kHz mono, converted to 128-channel mel-spectrograms, and processed in non-overlapping 30-second windows.AF-Whisper produces 50Hz features followed by stride-two pooling, with hidden dimension 1280.
- Audio Adaptor: Audio adaptor layers transform AF-Whisper hidden representations into embeddings that prompt the language model alongside textual instructions.The adaptor is denoted A(.) and produces a = A(ha).
- Language Model and TTS: AF3 uses Qwen-2.5-7B as its decoder-only causal language-model backbone and a streaming TTS module for voice-to-voice interaction.The TTS predicts subsequent audio tokens conditioned on incoming subword tokens and previously generated audio tokens.
4 Audio Flamingo 3 Training Data
AF3’s training data targets reasoning, long-context comprehension, controlled thinking, and multi-audio dialogue rather than recognition alone. Four curated datasets are integrated through a five-stage curriculum spanning short and long audio.
- Training Curriculum: AF3 uses five training stages, beginning with open recognition-focused datasets and adding four skill-focused datasets in later stages.Dataset mixtures use stage-specific combinations and weights.
- AudioSkills-XL: AudioSkills-XL expands short-audio reasoning data across sounds, music, and speech, including Speech-in-Sound QA for joint speech-and-ambient-scene reasoning.The expansion adds 3.5M sound and music QA pairs and 1M speech QA samples.
- LongAudio-XL: LongAudio-XL adds over 1M reasoning-focused QA examples from 30-second-to-10-minute speech to long-audio data previously focused on sounds and music.Sources include audiobooks, parliamentary debates, podcasts, calls, and natural dialogues assembled into coherent longer recordings.
- AF-Think: AF-Think provides 250k multiple-choice QA examples with short controlled thought prefixes and is upweighted during Stage 3.5 training.The prefixes average approximately 40 words and are generated from high-quality samples using ground-truth answers as guidance.
- AF-Chat: AF-Chat contains 75k multi-turn multi-audio dialogue instances curated from semantically related and unrelated audio clips.Dialogues average 4.6 audio clips and 6.2 turns, with later turns able to refer to earlier audios or responses.
5 Audio Flamingo 3 Training Strategy
AF3 uses a five-stage curriculum that progressively expands audio context, improves data quality, and diversifies tasks. Early stages align and tune the audio components using recognition-focused data with 30-second audio contexts.
- AF3’s five-stage strategy progressively increases audio context length, improves data quality, and diversifies tasks.
- Stages 1–2: Stages 1 and 2 use recognition-focused datasets while limiting audio context to 30 seconds.Stage 1 trains only the audio adaptor with the encoder and LLM frozen; Stage 2 fine-tunes the audio encoder and adaptor while keeping the LLM frozen.
6 Experiments
AF3 is evaluated against open-weight, open-source, and closed-source audio-language models across broad understanding, reasoning, chat, voice, and ablation benchmarks. It reports strong benchmark results, including multi-turn multi-audio gains, unified-encoder benefits, and faster speech generation.
- Experimental setup: AF3 is compared with recent open-weight, open-source, and closed-source LALMs across diverse audio benchmarks.The evaluation includes classification, audio QA, reasoning-focused QA, hallucination detection, captioning, ASR, and long-audio captioning and QA.
- Audio understanding and reasoning: AF3 sets new highs on MMAU (72.42), ClothoAQA (91.1), Clotho Entailment (92.9), and CMM Hallucination (86.7).It also outperforms Gemini 2.5 Pro on LongAudioBench and reports thinking-mode averages of 73.16 on MMAU-test and 74.26 on MMAU-test-mini.
- Chat evaluation: 30% relative improvement over Qwen2.5-Omni is reported for AF3-Chat on AF-Chat-test.The evaluation concerns extended dialog turns and references to multiple audio inputs.
- Voice-text and speech generation: AF3-Chat achieves WER 2.02 versus 2.72 for Qwen2.5-Omni and speaker similarity 0.61 versus 0.63.
- Voice-text and speech generation: A 10-second audio clip takes 6.68 seconds to produce, with 0.15-second time-to-first-token and 0.06-second inter-token latency.On an A100 GPU, AF3-Chat uses 5.94 seconds for text-to-audio token generation and 0.02 seconds for waveform synthesis.
- Ablation studies: AF-Whisper outperforms a CLAP-plus-Whisper-v3 dual-encoder setup under the same data budget.The comparison supports AF-Whisper as a single encoder for sound, music, and speech.
- Ablation studies: Removing AudioSkills-XL causes a significant performance drop, particularly on MMAU.The ablation compares the full setup with AudioSkills-XL removed from Stage 3.
7 Conclusion, Limitations and Future Work
The paper presents AF3 as an open audio-language model combining a custom Whisper encoder, novel data curation, and five-stage curriculum learning. It identifies cascaded voice chat, multilingual support, and reliance on closed-source synthetic data as future limitations to address.
- AF3 combines a custom Whisper encoder, novel data curation techniques, and a five-stage curriculum learning strategy.The paper also reports open-sourcing two large datasets and describes the architecture, training, inference, and evaluation pipeline.
- Limitations and future work: Future work targets reducing dependence on cascaded voice chat, adding multilingual support, and reducing reliance on closed-source models for synthetic data.
- AF-Whisper training used 128 NVIDIA A100 80GB GPUs, an effective batch size of 1024, AdamW, a 10^-4 learning rate, and fp16 precision.Training lasted five epochs over the complete dataset with random instance sampling for each batch.
A.2 Training Datasets
AF-Whisper training uses pooled audio-caption datasets and generates captions that describe speech, sounds, and music. Table 5 reports the statistics of these datasets.
- The training process generates transcripts, spoken-language characteristics, and audio captions for each dataset.
- When available, gold-standard metadata such as LibriSpeech transcripts or AudioCaps captions is incorporated into caption generation.
- Table 5 summarizes the audio-caption datasets used to train AF-Whisper.
B AudioSkills-XL
AudioSkills-XL organizes audio question answering around diverse reasoning skills spanning music, sound, and speech. Its examples combine perceptual cues, metadata, and contextual reasoning, with human evaluation procedures described for dialogue outputs.
- Dataset documentation: Table 6 details AudioSkills-XL’s reasoning types, source datasets, metadata, and prompts, while Table 9 and Table 10 report LongAudio-XL skill statistics and durations.
- Music reasoning skills: AudioSkills-XL covers music skills including genre, mood, melody, rhythm, instrumentation, performance, harmony, and temporal relations.
- Music reasoning skills: Music questions also assess texture, dynamics, functional context, lyrics, and historical or cultural context.
- Sound reasoning skills: Sound reasoning categories include speech-in-sound, eco-acoustic, acoustic-scene, and sound-based event reasoning.
- Illustrative questions: Examples include weather inference and interpretation of a speaker’s attitude from tone and ambient sounds.
- Evaluation: Human verification rates model outputs on factuality, usefulness, and depth using a 1–5 scale, with averages reported in Table 3.
E.2 Clustering for constructing AF-Chat
AF-Chat dialogues are constructed from semantically controlled audio pools rather than random clips. The clustering strategy combines similar and dissimilar audio to support coherent, diverse, and referential multi-turn conversations.
- Clustering procedure: Each seed audio is paired with its top 8 semantically similar and top 8 dissimilar clips using caption embeddings and FAISS search.
- Clustering procedure: Speech and environmental-sound clips come from Speech-in-Sound Caps, while music clips come from Music4All and the Million Song Dataset.
- Dialogue construction: The resulting 16-candidate pool supports conversations of up to 10 turns using combinations of related audio clips.
- Dialogue construction: The construction targets topical consistency, diversity and contrast, and clear references to earlier audio.
- Motivation: A preliminary human study found that participants naturally used highly similar or strongly contrasting clips during multi-audio conversations.
- Results: Clustered dialogues were more natural, coherent, and diverse than randomly constructed dialogues, and clustered training improved response relevance and conversational depth.
G AF3 Training Datasets
AF3’s training datasets are documented by scale, task content, and curriculum usage. Foundational datasets are converted into question-answer formats for consistent training across stages.
- Table 11 summarizes AF3 datasets by total hours, audio-QA pairs, and epochs used at each training stage.
- The dataset inventory includes foundational resources for captioning, classification, and related tasks.
- Foundational datasets are converted into QA formats using shared prompts for each task.
H AF3 Training Details
AF3 is trained through a five-stage curriculum, with stage-specific configurations detailed in Table 12.
- AF3 training is organized into five stages.Each stage uses specific training configurations.
- The training curriculum assigns distinct settings to each stage.The paper directs readers to Table 12 for these configurations.
- Table 12 provides the detailed training settings across stages.
I Streaming TTS System Architecture and Training Details
The streaming TTS system generates audio tokens autoregressively from AF3 text and prior audio history, then decodes them into voice output through a neural codec. Its causal, iterative design targets efficient low-latency speech streaming.
- Architecture: The TTS module predicts audio tokens from incoming subword text and previously generated audio tokens.The decoder-only architecture supports efficient streaming synthesis.
- Architecture: A neural audio codec decodes predicted tokens into voice output.The codec uses a fully causal convolutional architecture for streaming audio decoding.
- Codec: The codec encoder converts 44.1 kHz audio into a sequence approximately 4096 times shorter than the raw waveform.The output corresponds to approximately 10.8 frames per second.
- Codec: Residual Vector Quantization represents the encoded output using 72 RVQ levels.
- Codec: The decoder reconstructs waveform audio by symmetrically upsampling encoded representations and applying inverse STFT.
- Training: The codec is trained with adversarial training and mel-spectrogram reconstruction loss.
- Architecture: An MLP iteratively predicts progressively higher RVQ-token levels, using four inference iterations.The transformer conditions the MLP on text and previously generated audio-token states.
I.3 Training Data and Processing
AF3 training data combines large-scale speech, sound, and music resources with generated reasoning and dialogue supervision. The processing pipeline uses metadata, transcripts, captions, and targeted prompts to create diverse audio question-answering and chat tasks.
- Training data: Training speech data totals approximately 100,000 hours across LibriTTS-R, LibriHeavy, Multilingual LibriSpeech, and proprietary datasets.Speaker segments are randomly concatenated into samples lasting 1 to 120 seconds for long-form speech generation.
- LongAudio-XL: LongAudio-XL includes topic relationship, information extraction, ordering, causal, summarization, and emotional-state reasoning tasks.Examples include needle QA, topic QA, emotion flips, and causal reasoning.
- LongAudio-XL: Long-speech QA prompts require reasoning about specific details in transcripts from diverse speeches, lectures, stories, and parliamentary content.The prompts avoid naming speakers and target long-speech understanding.
- LongAudio-XL: Its emotional-state tasks ask models to identify emotions, explain causes, and characterize changes during conversations.Related prompts cover emotion flips, causal reasoning, sarcasm, and identification.
- AF-Chat: AF-Chat prompts generate multi-turn conversations about groups of speech, sound, and music samples using captions, transcripts, and speech characteristics.Music-focused chats ask questions requiring reasoning, world knowledge, and information extraction across samples.
- AudioSkills-XL: AudioSkills-XL prompts generate complex sound-reasoning and speech-in-sound QA, including indirectly phrased multiple-choice questions about sound events.The tasks require understanding audio and reasoning over contextual sound information.