Audio Flamingo 3: Advancing Audio Intelligence with Fully Open Large Audio Language Models

1 Introduction

• AF3 is reported to achieve state-of-the-art results on 20+ audio understanding and reasoning benchmarks, outperforming most open and closed LALMs while remaining fully open.The comparison names Qwen2.5-Omni as an open example and Gemini 2.5 Pro as a closed example.Source: Figure 1 · p. 1, §1 · pp. 1–2, Table 1 · p. 2
• The introduction identifies expert-level reasoning [101], multi-turn and multi-audio dialogue [44], and long-audio understanding as gaps, noting that models often prioritize short recognition tasks over complex skills.It also says audio comprehension remains limited in accessibility and capability [54] [106].Source: §1 · pp. 1–2
• AF3 adds long-context audio QA up to 10 minutes, on-demand concise reasoning, and AF3-Chat for multi-turn, multi-audio chat and voice-to-voice interaction.The long-context QA extends beyond sounds to include speech.Source: §1 · p. 2
• The four proposed datasets comprise AudioSkills-XL with 8M AQA pairs, LongAudio-XL with 1.25M long-audio QA pairs, AF-Chat with 75k instances, and AF-Think with 250k+ AQA pairs.AF-Think uses short length prefixes to encourage chain-of-thought-style reasoning before answers; AF-Chat is curated with a novel algorithm.Source: §1 · p. 2
• AF3’s enabling methods include AF-Whisper, a unified encoder trained for general-purpose representations across speech, sounds, and music, and a five-stage curriculum that increases context length and task complexity.The encoder is pretrained using a novel strategy on large-scale audio-caption pairs.Source: §1 · p. 2
• The authors define fully open as publicly releasing model weights, training data, and code with training-method transparency, while limiting releases to research use because of data licensing and scope.They also state that code, training recipes, and four new datasets will be released.Source: §1 · p. 2

Audio Language Models

• ALMs include joint audio-text embedding encoders such as CLAP, Wav2CLIP, and AudioCLIP, alongside LALMs that pair audio encoders with decoder-only LLMs.LALMs have improved ASR, audio captioning, and acoustic scene classification, and enabled open-ended audio question answering.Source: §2 · p. 3
• Current LALMs lack multi-turn, multi-audio chat, long-context comprehension, unified speech-sound-music understanding, and full release of training code and data.The limited transparency of partially open models impedes reproducibility and scientific progress.Source: §2 · p. 3

Reasoning and Long-Context Understanding

• AF2 [40] was the first step toward long-context audio comprehension, but its coverage was limited to sounds and music.Long-context video modeling had also advanced in the vision-language domain [17].Source: §2 · p. 3
• AF3 combines controlled reasoning supervision, long-context training, and modality diversity to support reasoning and long-context comprehension that includes speech.Related efforts advanced reasoning through reasoning datasets [101] [110], multimodal perception [116] [105], and chain-of-thought prompting [80].Source: §2 · p. 3

3.1 Audio Flamingo 3 Architecture

• AF3 combines AF-Whisper, audio adaptor layers, Qwen-2.5-7B, and streaming TTS for voice-to-voice interaction.The adaptor maps encoded audio features into embeddings used as prompts alongside the textual instruction. The decoder-only TTS predicts audio tokens from incoming LLM subword tokens and previously generated audio tokens, supporting streaming inputs and outputs.Source: §3.1 · pp. 3–4
• AF-Whisper unifies speech, sound, and music in one encoder, avoiding separate modality encoders that increase complexity, cause frame-rate mismatches, and risk training instability.The encoder starts from pre-trained Whisper large-v3 and is trained with an audio-captioning next-token-prediction objective; GPT-4.1 generates captions using available metadata, with AF2 or Whisper-Large-v3 ASR synthesizing missing metadata.Source: §3.1 · p. 3
• Audio inputs are resampled to 16kHz mono, converted to 128-channel mel-spectrograms, and processed in non-overlapping 30-second chunks.AF-Whisper outputs features at 50Hz, followed by stride-two pooling; feature dimension d is 1280, while temporal resolution depends on audio length and the training-stage window limit.Source: §3.1 · p. 4

4.1 AudioSkills-XL: Expanding AudioSkills with Reasoning-Focused QAs

• AudioSkills-XL expands AudioSkills by 4.5M QA pairs to 8M, adding reasoning-focused sound, music, and speech tasks for short clips (≤30s).The curation adds 3.5M sound and music QA pairs and 1M speech QA samples; it uses music metadata to generate complex QAs and speech-aware captions to represent both speech and ambient sounds.Source: §4.1 · pp. 4–5

4.2 LongAudio-XL: Expanding LongAudio with Long Speech QA

• LongAudio-XL adds over 1M reasoning-focused QA examples for long-form speech from 30 seconds to 10 minutes.It merges consecutive short segments into coherent longer audio and covers sarcasm, emotion, topic relationships, information extraction, summarization, and temporal ordering and grounding.Source: §4.2 · pp. 5–6

4.3 AF-Think: Towards flexible, on-demand reasoning

• AF-Think contains 250k MCQ-based QA pairs with short thought prefixes averaging approximately 40 words, enabling reasoning only when prompted.In Stage 3.5, AF-Think is upweighted relative to standard QA data; Gemini 2.0 Flash generates the prefixes from selected AudioSkills-XL and LongAudio-XL questions using the correct answer as guidance.Source: §4.3 · p. 6

4.4 AF-Chat: Multi-turn Multi-audio Chat Data

• AF-Chat provides 75k multi-turn, multi-audio dialogue instances, averaging 4.6 clips and 6.2 turns per dialogue.Curators select semantically similar and dissimilar clips to ground GPT-4.1-generated conversations, which can refer back to earlier audio; the 200-example AF-Chat-test set uses clips unseen during training.Source: §4.4 · pp. 6–7

Stages 1–2: Alignment and Encoder Tuning

• Stages 1–2 train the adaptor and then the audio encoder while freezing the LLM, using recognition-focused data and audio contexts limited to 30 seconds.Stage 1 aligns encoder representations with the language model; Stage 2 adapts AF-Whisper to diverse datasets to broaden audio understanding.Source: §5 · p. 7

Stage 3: Full Fine-Tuning

• Stage 3 fine-tunes the full model for reasoning and skill acquisition with foundational, QA, and AudioSkills-XL data, extending audio context to 2.5 minutes.Short audios remain central because skill-specific data is easy to scale on them; the stage-end model is called AF3.Source: §5 · p. 7

Stage 3.5: Context Extension and Thinking

• Stage 3.5 adds LongAudio-XL and AF-Think and trains LLM LoRA adapters with original model weights frozen to extend context and encourage CoT-style reasoning on demand.This stage also retains the Stage 3 data mixture.Source: §5 · p. 7

Stage 4: Chat and Voice Fine-Tuning

• Stage 4 fine-tunes the entire model on AF-Chat for multi-turn interactive voice dialogue, conversational audio understanding, and response generation; its resulting model is AF3-Chat.The five-stage strategy progressively enhances capabilities by increasing context length, improving data quality, and diversifying tasks.Source: §5 · p. 7

6 Experiments

• AF3 is evaluated across audio classification, QA and reasoning, hallucination detection, captioning, ASR, long-audio tasks, chat, voice-text, and zero-shot TTS benchmarks.Baseline results are based on the authors’ own evaluations; voice-to-voice evaluation is outside scope, and Table 3 compares only open LALMs.Source: §6 · pp. 7–9, Table 3 · p. 9

6.1 Audio Understanding and Reasoning Evaluation

• AF3 surpasses prior open-weight and closed-source models across benchmarks, setting highs on MMAU (72.42), ClothoAQA (91.1), Clotho Entailment (92.9), and CMM Hallucination (86.7).It outperforms Gemini 2.5 Pro on LongAudioBench and achieves state-of-the-art ASR results on LibriSpeech, SPGISpeech, and VoxPopuli. With the specified thinking prompt, it reports 73.16 on MMAU-test and 74.26 on MMAU-test-mini; thinking is activated after Stage 3.5, and the checkpoint remains usable without it.Source: Table 2 · p. 8, §6.1 · p. 9

6.2 Chat and TTS Evaluation

• AF3-Chat surpasses Qwen2.5-Omni across OpenAudioBench, performs comparably on VoiceBench, and records SEED WER 2.02 versus 2.72 and speaker similarity 0.61 versus 0.63.VoiceBench comparisons include Qwen2.5-Omni and Qwen2-Audio Chat; it evaluates AdvBench, CommonEval, and OpenBookQA.Source: Table 3 · p. 9, §6.2 · pp. 9–10
• On AF-Chat-test, AF3-Chat improves relatively by 30% over Qwen2.5-Omni in multi-turn, multi-audio chat.The evaluation indicates effective handling of extended dialogue turns, deeper contextual reasoning, and more accurate references to multiple audio inputs.Source: §6.2 · p. 9
• For 10-second generation on A100, AF3-Chat uses 5.94 seconds for token generation and 0.02 seconds for synthesis, versus Qwen2.5-Omni Talker’s 14.62 and 1.26 seconds.Streaming TTS reaches 0.15-second time-to-first-token and 0.06-second inter-token latency, including waveform synthesis, producing a 10-second clip in 6.68 seconds.Source: §6.2 · p. 10

6.3 Ablation Studies

• In 10%-data ablations, AF-Whisper beats dual CLAP/Whisper-v3 encoders, while removing Stage 3 AudioSkills-XL causes a marked drop, especially on MMAU.The encoder comparison uses the same data budget; the results support AF-Whisper as a unified encoder and AudioSkills-XL’s contribution to generalization and robustness.Source: Table 4 · p. 10, §6.3 · p. 10

7 Conclusion, Limitations and Future Work

• Audio Flamingo 3 achieves state-of-the-art performance in audio understanding and reasoning.The paper describes it as the most capable and open large audio-language model.Source: §7 · p. 10
• AF3 introduces multi-turn multi-audio chat, on-demand thinking, and voice chat.Source: §7 · p. 10
• AF3 uses a custom Whisper, novel data curation techniques, and a five-stage curriculum learning strategy.Source: §7 · p. 10
• The authors detail the architecture, training, inference, and evaluation pipeline, and open-source two large datasets.Source: §7 · p. 10
• Future work targets eliminating cascaded voice chat, enabling multilingual AF3, and reducing reliance on closed-source models for synthetic data.Source: §7 · p. 10

A.1 Training Details

• AF-Whisper was trained for five epochs on the complete dataset using 128 NVIDIA A100 80GB GPUs and an effective batch size of 1024.Training used AdamW with learning rate 10−4 and weight decay 0.1, fp16 precision, and random sampling from the full instance pool for each batch.Source: §A.1 · p. 19

A.2 Training Datasets

• For each dataset, the annotation process generates transcripts, spoken-language characteristics, and audio captions, using gold-standard metadata when available.Examples include LibriSpeech transcripts and AudioCaps captions.Source: §A.2 · p. 19
• GPT-4.1 produces final captions in a format modeled on Figure 50, while AF2 extracts speech characteristics using a prompt about pitch, emotion, mood, speed, and other dynamics.The caption prompt uses a modified exemplar.Source: §A.2 · p. 19
• Table 5 reports statistics for the audio-caption datasets used to train AF-Whisper.The supplied passage does not provide the table’s numerical values.Source: Table 5 · p. 19

B AudioSkills-XL

• AudioSkills-XL’s statistics table organizes examples by reasoning type and documents source datasets, metadata, and QA-generation prompts; ungrayed rows mark this paper’s contributions.Speech QA types are shared with LongAudio-XL and explained in Section 4.2.Source: Table 6 · p. 20, B AudioSkills-XL · p. 20

B.1.1 Music Reasoning

• Music reasoning questions address genre and style, mood, temporal development, lyrical themes, melody, and rhythm and tempo.These skills range from interpreting musical cues to understanding changes in structure over time.Source: §B.1.1 · p. 20

B.1.2 Music Knowledge

• Music knowledge covers instrumentation, performance, sound texture, metre and rhythm, melody, and dynamics and expression.The skills attend to musical elements and delivery, including shifts in dynamics and expressive technique.Source: §B.1.2 · p. 21

B.1.3 Sound Reasoning

• Sound reasoning includes speech alongside ambient audio, environmental inference from acoustic cues, scene interpretation, and reasoning about sound-based events.Examples of these reasoning tasks are shown in Figures 5–7.Source: §B.1.3 · pp. 21–22

C LongAudio-XL

• LongAudio-XL documentation provides skill-wise statistics, source datasets, and minimum, maximum, and average sample durations.The appendix also includes examples of LongAudio-XL.Source: C LongAudio-XL · p. 23, Figure 8 · p. 23

D AF-Think

• AF-Think’s speech QA examples often require reasoning about ambient sounds as well as spoken content, which the appendix says increases task complexity and demands deeper inference.Table 7 presents detailed statistics for AF-Think.Source: Table 7 · p. 23

E AF-Chat

• AF-Chat documentation includes detailed statistics and two examples, one each for sound and music.The examples are presented in Figures 10 and 11.Source: Table 8 · p. 24, E AF-Chat · p. 24

E.1 Human Study for AF-Chat-test

• Four Ph.D. students proficient in audio research rated each AF-Chat-test dialogue instance for factuality, usefulness, and depth on a 1–5 scale; author responses were scored for reference.The final Table 3 score averages ratings across all instances, and the study was approved by the institution’s IRB.Source: §E.1 · p. 24

E.2 Clustering for constructing AF-Chat

• For each seed, the method retrieves the top 8 similar and 8 dissimilar clips using captions, NV-Embed-v2 embeddings, and FAISS similarity search [31].Source: §E.2 · p. 24
• GPT-4.1 constructs conversations of up to 10 turns from the 16-clip pool, sourced from Speech-in-Sound Caps for speech and environmental audio and Music4All and MSD for music.Source: §E.2 · pp. 24, 26
• The design targets topical consistency, contrastive diversity, and references to earlier audio; a preliminary human study found participants gravitated toward similar or strongly contrasting clips.Source: §E.2 · p. 26
• Clustered AF3-Chat training produced more natural, coherent, diverse dialogues and outperformed random-pool training in response relevance and conversational depth.Source: §E.2 · p. 26

G AF3 Training Datasets

• Table 11 summarizes training datasets’ hours, audio-QA pair counts, and epochs per stage; foundational datasets are converted to QA using prompts from.Source: G AF3 Training Datasets · p. 27

• The section presents training settings across all five stages and directs readers to Table 12 for details.Source: H AF3 Training Details · p. 27

I Streaming TTS System Architecture and Training Details

• AF3 generates audio tokens autoregressively from incoming subword text and prior audio tokens, then decodes them into voice output with a neural audio codec.The decoder-only design simplifies the pipeline and minimizes latency for real-time streaming; Figure 13 also depicts iterative audio-token unmasking.Source: I Streaming TTS System Architecture… · p. 27, Figure 13 · p. 28

I.1 Neural Audio Codec

• The fully causal codec uses RVQ and compresses audio to about 10.8 frames per second for streaming.Audio is resampled to 44.1 kHz and represented with STFT parameters (hop size 8, window size 32); the encoder output is 4096 times shorter than the waveform, uses 72 RVQ levels, and is decoded through a symmetric transposed-convolution architecture and iSTFT. Training combines adversarial training with mel-spectrogram reconstruction loss; the codec follows [61], its 1D-ConvNeXt blocks follow, and its iSTFT is similar to Vocos, while RVQ follows [69].Source: §I.1 · pp. 27–28

I.2 Text-to-Speech (TTS) Module

• The TTS transformer conditions on text and generated audio history, while an MLP predicts progressively higher RVQ levels in four inference iterations.Audio tokens are generated whenever AF3 emits a text token, without explicit text-speech alignments. Training uses teacher-forced ground-truth audio tokens and a 1024-mixture Gaussian objective for cumulative RVQ embeddings [61]; the decoder resembles DiT-XL [93] [70] and the MLP has 3 layers totaling 644 M parameters.Source: §I.2 · pp. 28–29

I.3 Training Data and Processing

• TTS training uses about 100,000 hours of audio and concatenates same-speaker segments into 1–120-second samples for long-form generation.The data includes LibriTTS-R [64], LibriHeavy [59], the English portion of Multilingual LibriSpeech [95], and proprietary datasets.Source: §I.3 · p. 29

J Qualitative Examples

• On an unseen Superman 2025 trailer clip, AF3 identifies background sounds, spoken content, speaker turns, and transcriptions.The example demonstrates audio captioning on a clip captured in the wild.Source: Figure 14 · p. 29
• The qualitative examples include prompts for LongAudio-XL QA, AudioSkills-XL music and sound reasoning, AF-Chat dialogues, and AF-Think chain-of-thought reasoning.The shown prompts include topic-relationship QA, music-texture reasoning, eco-acoustic reasoning, multi-turn speech-and-sound chat, and music-focused CoT-style reasoning.Source: Figure 35 · p. 52, Figure 40 · p. 55, Figure 43 · p. 56