Source-linked AI summary
Audio Flamingo Next: Next-Generation Open Audio-Language Models for Speech, Sound, and Music
Sreyan Ghosh, Arushi Goel, Kaousheik Jayakumar, Lasha Koroshinadze, Nishit Anand, Zhifeng Kong, Siddharth Gururani, Sang-gil Lee, Jaehyeon Kim, Aya Aljafari, Chao-Han Huck Yang, Sungwon Kim, Ramani Duraiswami, Dinesh Manocha, Mohammad Shoeybi, Bryan Catanzaro, Ming-Yu Liu, Wei Ping
TL;DR
Open audio-language models have limited benchmark coverage and weak support for long, diverse audio. AF-Next addresses these gaps with scaled data, temporal reasoning, and long-context training, achieving strong performance across audio tasks while leaving robust distant-evidence reasoning as an open challenge.
Problem
Open audio-language models are constrained by narrow academic benchmarks and limited robustness for long, noisy, and diverse real-world audio.
Method
AF-Next combines internet-scale data curation, capability expansion, Temporal Audio Chain-of-Thought, long-context training, and three released model variants.
Results
AF-Next achieves state-of-the-art performance across 20+ benchmarks, substantially outperforms similarly sized open models, and remains competitive with larger open-weight and closed models.
Takeaways & Limitations
AF-Next supports long-form audio up to 30 minutes, timestamped captioning, multilingual ASR, and open research through released code, data, techniques, and checkpoints.
Takeaways & Limitations
Robust long-context reasoning remains challenging when evidence is temporally distant, sparse, or distributed across multiple segments.
Abstract
from arXiv · showhide
We present Audio Flamingo Next (AF-Next), the next-generation and most capable large audio-language model in the Audio Flamingo series, designed to advance understanding and reasoning over speech, environmental sounds and music. Compared to Audio Flamingo 3, AF-Next introduces: (i) a stronger foundational audio-language model that significantly improves accuracy across diverse audio understanding tasks; (ii) scalable strategies for constructing large-scale audio understanding and reasoning data beyond existing academic benchmarks; (iii) support for long and complex audio inputs up to 30 minutes; and (iv) Temporal Audio Chain-of-Thought, a new reasoning paradigm that explicitly grounds intermediate reasoning steps to timestamps in long audio, enabling fine-grained temporal alignment and improved interpretability. To enable these capabilities, we first conduct a systematic analysis of Audio Flamingo 3 to identify key gaps in audio understanding and reasoning. We then curate and scale new large-scale datasets totaling over 1 million hours to address these limitations and expand the existing AudioSkills-XL, LongAudio-XL, AF-Think and AF-Chat datasets. AF-Next is trained using a curriculum-based strategy spanning pre-training, mid-training and post-training stages. Extensive experiments across 20 audio understanding and reasoning benchmarks, including challenging long-audio tasks, show that AF-Next outperforms similarly sized open models by large margins and remains highly competitive with and sometimes surpasses, much larger open-weight and closed models. Beyond benchmark performance, AF-Next exhibits strong real-world utility and transfers well to unseen tasks, highlighting its robustness and generalization ability. In addition to all data, code and methods, we open-source 3 variants of AF-Next, including AF-Next-Instruct, AF-Next-Think and AF-Next-Captioner.
1. Introduction
AF-Next targets limited coverage and weak real-world generalization in open audio-language models by scaling data, capabilities, and temporally grounded reasoning. It achieves broad benchmark strength while releasing an open training recipe and three specialized variants.
- Open LALM development remains constrained by closed systems and narrow academic benchmarks that incompletely cover diverse audio.
- AF-Next scales training beyond academic datasets using long, diverse, and acoustically challenging internet-scale audio.
- AF-Next adds multi-talker ASR, timestamped prediction, long-form captioning, and instruction following to the Audio Flamingo task suite.
- Temporal Audio Chain-of-Thought grounds intermediate reasoning steps to timestamps in long audio.
- Across 20+ benchmarks, AF-Next outperforms similarly sized open models by large margins and sometimes surpasses much larger open-weight and closed models.
- The project releases training and inference code plus AF-Next-Instruct, AF-Next-Think, and AF-Next-Captioner checkpoints.
2. Related Works
Related work spans multimodal audio-language architectures, increasingly scaled audio data, and chain-of-thought reasoning, while transparency and long-audio coverage remain important concerns.
- LALMs use decoder-only language models augmented with audio encoders, following earlier encoder-only audio-language models for shared embedding and retrieval.
- Representative LALMs include Audio Flamingo, Qwen-Audio, SALMONN, Pengi, GAMA, and related systems.
- Audio understanding has pursued aggressive data scaling, but opaque training data limits transparency and reproducibility for many systems.
- Recent audio chain-of-thought methods improve deliberate reasoning, but are largely designed for short audio with easily accessible evidence.
3. Methodology
AF-Next combines a multimodal language-model architecture with expanded real-world training data, temporal reasoning, and long-context parallel training. Its design targets diverse skills and the computational cost of long audio.
- Architecture: AF-Next comprises a sliding-window audio encoder, audio projector, text-only LLM backbone, and streaming TTS module.
- Architecture: The AF-Whisper encoder processes multilingual speech and multi-talker ASR data, while a 2-layer MLP maps audio representations into the LLM embedding space.
- Architecture: RoTE uses absolute token timestamps rather than discrete positions, supporting temporally grounded representations for long-form audio reasoning.
- Data Curation: AF-Next curates data to address skill and distributional gaps, including music, multilingual ASR, multi-talker speech, long captioning, multi-audio, chat, and safety.
- Data Curation: Temporal Audio Chain-of-Thought grounds intermediate reasoning steps to timestamps for complex long recordings with overlapping and dispersed events.
- Long-Context Training: Long-audio training addresses sequence expansion and quadratic attention memory through sequence-level packing and hybrid sequence parallelism across GPUs.
4. Experiments
The experiments evaluate three AF-Next variants against open, open-weight, and closed LALMs across diverse audio, reasoning, speech, and long-audio benchmarks using accuracy, WER, and GPT evaluation.
- Experimental Setup: AF-Next is evaluated through pre-training, mid-training, post-training, and chain-of-thought training on 128 NVIDIA H100 GPUs.
- Baselines: AF-Next-Instruct is evaluated against recent systems including GAMA, Audio Flamingo models, Qwen variants, R1-AQA, Pengi, Phi-4-mm, and others.
- Evaluation: The benchmark suite covers audio classification, reasoning-focused audio QA, hallucination detection, ASR, long-audio understanding, and music captioning.
- Evaluation: Table 1 reports WER, accuracy, and GPT evaluation while comparing AF-Next with prior LALMs and highlighting closed-source, open-weight, and open-source models.
- Additional Evaluation: Table 2 evaluates AF-Next-Instruct on VoiceBench and speech translation benchmarks, extending the comparison beyond the Table 1 task suite.
- Metrics: Accuracy is computed by exact matching or CLAP-based retrieval, while verbose multiple-choice outputs are normalized with regex extraction.
5. Results
AF-Next delivers broad gains across audio understanding, music, long-audio reasoning, speech recognition, voice understanding, and speech translation. Its variants provide additional benefits on challenging reasoning and captioning tasks.
- Audio understanding and reasoning: AF-Next-Instruct substantially outperforms prior models across diverse audio understanding and reasoning benchmarks, with AF-Next-Think and AF-Next-Captioner adding further gains.The reported results span general audio, music, long-context audio, speech recognition, voice understanding, and translation.
- Audio understanding and reasoning: 75.76 average accuracy is achieved on MMAU-v05.15.25 by AF-Next-Captioner, exceeding Audio Flamingo 3’s 72.42.Gains extend across sound, music, and speech subcategories.
- Music understanding: 8.8 coverage and 8.9 correctness are achieved on SongCaps, improving over AF3’s 6.7 and 6.2 for music captioning.AF-Next also reports strong music classification and instrument-recognition results.
- Long audio understanding: 73.9 on LongAudioBench and 81.2 on its +Speech variant exceed AF3 and Gemini 2.5 Pro, demonstrating strong long-context audio reasoning.The comparisons are 73.9 versus 68.6 and 60.4, and 81.2 versus 72.9 and 66.2, respectively.
- Automatic speech recognition: 1.54 WER on LibriSpeech test-clean and 2.76 on test-other establish new lows among LALMs, while AF-Next remains competitive across additional English ASR benchmarks.It also achieves the best WER on Common Voice 15, GigaSpeech, and VoxPopuli.
- Voice understanding and speech translation: 21.9 BLEU for EN→X Arabic translation improves over Phi-4-mm’s 9.9, while AF-Next leads on Chinese and remains competitive on Japanese and German.For X→EN translation, it also leads on Chinese and Arabic.
6. Conclusion
AF-Next combines broad audio capabilities with stronger long-form reasoning and real-world robustness. The paper releases its code, checkpoints, and core techniques, including Temporal Audio Chain-of-Thought for timestamp-grounded reasoning.
- Conclusion: AF-Next supports audio up to 30 minutes, multi-turn chat, timestamped captioning, and multilingual ASR while demonstrating stronger robustness in real-world use cases.These capabilities accompany state-of-the-art performance across contemporary audio understanding benchmarks.
- Conclusion: Temporal Audio Chain-of-Thought grounds intermediate evidence in time for long-audio question answering, enabling more faithful and robust reasoning.The paradigm explicitly links reasoning steps to timestamps.
- Conclusion: The authors open-source training code, model checkpoints, and core techniques to support future research in open audio-language modeling.The released model variants include AF-Next-Instruct, AF-Next-Think, and AF-Next-Captioner.
Limitations
AF-Next’s limitations concern uneven internet-scale data coverage, difficult reasoning over long contexts, and evaluation that does not fully cover several supported capabilities.
- Data coverage: Internet-scale audio remains noisy and unevenly distributed across domains, languages, and acoustic conditions.Low-resource languages, rare sound events, and specialized real-world domains are underrepresented.
- Data coverage: Future work should improve the diversity, balance, and coverage of open audio datasets.
- Long-context reasoning: Robust reasoning remains challenging when relevant evidence is temporally distant, sparse, or distributed across multiple audio segments.This limitation persists despite support for audio inputs up to 30 minutes.
- Long-context reasoning: Temporal Audio Chain-of-Thought improves temporal grounding, but stronger long-context memory, retrieval, and evidence aggregation remain needed.
- Evaluation coverage: Evaluation focuses on established benchmarks and does not yet fully cover multi-talker ASR, speaker diarization, timestamped captioning, or voice-to-voice interaction.Broader evaluation protocols for these capabilities are identified as an important next step.
B. Potential Risks
AF-Next inherits risks associated with large language models and may reflect biases in its internet-scale audio and text training distribution.
- Model risks: AF-Next may generate toxic, biased, or hallucinated outputs, including factually incorrect responses for some long-audio inputs.The paper notes varying recognition accuracy across accents and dialects and difficulty aggregating evidence over extended temporal spans.
- Mitigation: 386K safety and instruction-following fine-tuning samples teach refusal and abstention behavior to mitigate these risks.
C. Data Anonymization and Privacy Safeguards
The paper derives training data from public academic datasets and openly accessible internet audio while applying safeguards against linking samples to speaker identities.
- Data sources: The training data comes from publicly available academic datasets and openly accessible internet audio.The paper states that it does not intentionally collect data naming or uniquely identifying individuals.
- Privacy safeguards: Metadata that could link audio samples to real-world speaker identities is stripped during data curation.
- Privacy safeguards: 386K safety fine-tuning samples train the model to refuse requests for personally identifiable information.
D. AF-Next Training Details
AF-Next training is described across multiple stages, with detailed configurations provided in Table 3.
- Training settings: The model’s training settings are presented across all stages, with each stage using specific configurations.
- Training settings: Table 3 summarizes the training settings across stages.
E. AF-Next Training Dataset Details
AF-Next’s training data are organized across pre-training and fine-tuning datasets, with composition tracked by dataset size, audio-QA pairs, and training epochs. The data-generation examples cover long-audio question answering, captioning, temporal reasoning, multi-speaker recognition, multi-turn chat, and safety responses.
- Dataset composition: Table 4 records each training dataset’s total hours, audio-QA pairs, and training epochs across the training stages.The datasets include both pre-training and fine-tuning data.
- Dataset composition: Foundational datasets such as captioning and classification are converted into question-answer formats using task-specific prompts.The same prompt sets are used for each task, following prior work cited in the paper.
- Long-audio data: Long-audio data generation targets multi-turn chat, counting, detailed captioning, needle-in-the-haystack retrieval, subscene captioning, and temporal understanding.These prompt families cover several complementary forms of long-context audio reasoning and description.
- Specialized training data: Additional examples train timestamped captions, diarized multi-speaker transcripts, multi-turn audio chat, and safety-aligned refusal responses.The examples span music, speech, sports, documentaries, commentary, benign questions, and harmful requests.