PaperlayerView exact revision on arXiv

Source-linked AI summary

Audio Flamingo Next: Next-Generation Open Audio-Language Models for Speech, Sound, and Music

Sreyan Ghosh, Arushi Goel, Kaousheik Jayakumar, Lasha Koroshinadze, Nishit Anand, Zhifeng Kong, Siddharth Gururani, Sang-gil Lee, Jaehyeon Kim, Aya Aljafari, Chao-Han Huck Yang, Sungwon Kim, Ramani Duraiswami, Dinesh Manocha, Mohammad Shoeybi, Bryan Catanzaro, Ming-Yu Liu, Wei Ping

arXiv:2604.10905v1cs.SDcs.AIcs.CLeess.AS

TL;DR

Open audio-language models have limited benchmark coverage and weak support for long, diverse audio. AF-Next addresses these gaps with scaled data, temporal reasoning, and long-context training, achieving strong performance across audio tasks while leaving robust distant-evidence reasoning as an open challenge.

  • Problem

    Open audio-language models are constrained by narrow academic benchmarks and limited robustness for long, noisy, and diverse real-world audio.

    Source: §1 · p. 1

  • Method

    AF-Next combines internet-scale data curation, capability expansion, Temporal Audio Chain-of-Thought, long-context training, and three released model variants.

    Source: §1 · p. 2, §3.1 · p. 4, §3.3 · p. 8

  • Results

    AF-Next achieves state-of-the-art performance across 20+ benchmarks, substantially outperforms similarly sized open models, and remains competitive with larger open-weight and closed models.

    Source: §1 · p. 2, §6 · p. 11

  • Takeaways & Limitations

    AF-Next supports long-form audio up to 30 minutes, timestamped captioning, multilingual ASR, and open research through released code, data, techniques, and checkpoints.

    Source: §1 · p. 2, §6 · p. 11

    Robust long-context reasoning remains challenging when evidence is temporally distant, sparse, or distributed across multiple segments.

    Source: Limitations · p. 12

Abstract

from arXiv · show

We present Audio Flamingo Next (AF-Next), the next-generation and most capable large audio-language model in the Audio Flamingo series, designed to advance understanding and reasoning over speech, environmental sounds and music. Compared to Audio Flamingo 3, AF-Next introduces: (i) a stronger foundational audio-language model that significantly improves accuracy across diverse audio understanding tasks; (ii) scalable strategies for constructing large-scale audio understanding and reasoning data beyond existing academic benchmarks; (iii) support for long and complex audio inputs up to 30 minutes; and (iv) Temporal Audio Chain-of-Thought, a new reasoning paradigm that explicitly grounds intermediate reasoning steps to timestamps in long audio, enabling fine-grained temporal alignment and improved interpretability. To enable these capabilities, we first conduct a systematic analysis of Audio Flamingo 3 to identify key gaps in audio understanding and reasoning. We then curate and scale new large-scale datasets totaling over 1 million hours to address these limitations and expand the existing AudioSkills-XL, LongAudio-XL, AF-Think and AF-Chat datasets. AF-Next is trained using a curriculum-based strategy spanning pre-training, mid-training and post-training stages. Extensive experiments across 20 audio understanding and reasoning benchmarks, including challenging long-audio tasks, show that AF-Next outperforms similarly sized open models by large margins and remains highly competitive with and sometimes surpasses, much larger open-weight and closed models. Beyond benchmark performance, AF-Next exhibits strong real-world utility and transfers well to unseen tasks, highlighting its robustness and generalization ability. In addition to all data, code and methods, we open-source 3 variants of AF-Next, including AF-Next-Instruct, AF-Next-Think and AF-Next-Captioner.

1. Introduction

AF-Next targets limited coverage and weak real-world generalization in open audio-language models by scaling data, capabilities, and temporally grounded reasoning. It achieves broad benchmark strength while releasing an open training recipe and three specialized variants.

  • Open LALM development remains constrained by closed systems and narrow academic benchmarks that incompletely cover diverse audio.Source: §1 · pp. 1–2
  • AF-Next scales training beyond academic datasets using long, diverse, and acoustically challenging internet-scale audio.Source: §1 · p. 2
  • AF-Next adds multi-talker ASR, timestamped prediction, long-form captioning, and instruction following to the Audio Flamingo task suite.Source: §1 · p. 2
  • Temporal Audio Chain-of-Thought grounds intermediate reasoning steps to timestamps in long audio.Source: §1 · p. 2
  • Across 20+ benchmarks, AF-Next outperforms similarly sized open models by large margins and sometimes surpasses much larger open-weight and closed models.Source: §1 · p. 2
  • The project releases training and inference code plus AF-Next-Instruct, AF-Next-Think, and AF-Next-Captioner checkpoints.Source: §1 · p. 2

2. Related Works

Related work spans multimodal audio-language architectures, increasingly scaled audio data, and chain-of-thought reasoning, while transparency and long-audio coverage remain important concerns.

  • LALMs use decoder-only language models augmented with audio encoders, following earlier encoder-only audio-language models for shared embedding and retrieval.Source: §2 · p. 3
  • Representative LALMs include Audio Flamingo, Qwen-Audio, SALMONN, Pengi, GAMA, and related systems.Source: §2 · p. 3
  • Audio understanding has pursued aggressive data scaling, but opaque training data limits transparency and reproducibility for many systems.Source: §2 · pp. 2–3
  • Recent audio chain-of-thought methods improve deliberate reasoning, but are largely designed for short audio with easily accessible evidence.Source: §2 · p. 3

3. Methodology

AF-Next combines a multimodal language-model architecture with expanded real-world training data, temporal reasoning, and long-context parallel training. Its design targets diverse skills and the computational cost of long audio.

Architecture

  • AF-Next comprises a sliding-window audio encoder, audio projector, text-only LLM backbone, and streaming TTS module.Source: §3.1 · p. 3
  • The AF-Whisper encoder processes multilingual speech and multi-talker ASR data, while a 2-layer MLP maps audio representations into the LLM embedding space.Source: §3.1 · pp. 3–4
  • RoTE uses absolute token timestamps rather than discrete positions, supporting temporally grounded representations for long-form audio reasoning.Source: §3.1 · p. 4

Data Curation

  • AF-Next curates data to address skill and distributional gaps, including music, multilingual ASR, multi-talker speech, long captioning, multi-audio, chat, and safety.Source: §3.2.1 · pp. 5–6
  • Temporal Audio Chain-of-Thought grounds intermediate reasoning steps to timestamps for complex long recordings with overlapping and dispersed events.Source: §3.2.1 · p. 6

Long-Context Training

  • Long-audio training addresses sequence expansion and quadratic attention memory through sequence-level packing and hybrid sequence parallelism across GPUs.Source: Figure 3 · p. 5, §3.3 · p. 8

4. Experiments

The experiments evaluate three AF-Next variants against open, open-weight, and closed LALMs across diverse audio, reasoning, speech, and long-audio benchmarks using accuracy, WER, and GPT evaluation.

Experimental Setup

  • AF-Next is evaluated through pre-training, mid-training, post-training, and chain-of-thought training on 128 NVIDIA H100 GPUs.Source: §4 · p. 8

Baselines

  • AF-Next-Instruct is evaluated against recent systems including GAMA, Audio Flamingo models, Qwen variants, R1-AQA, Pengi, Phi-4-mm, and others.Source: §4 · p. 8

Evaluation

  • The benchmark suite covers audio classification, reasoning-focused audio QA, hallucination detection, ASR, long-audio understanding, and music captioning.Source: §4 · pp. 8–9
  • Table 1 reports WER, accuracy, and GPT evaluation while comparing AF-Next with prior LALMs and highlighting closed-source, open-weight, and open-source models.Source: Table 1 · p. 9

Additional Evaluation

  • Table 2 evaluates AF-Next-Instruct on VoiceBench and speech translation benchmarks, extending the comparison beyond the Table 1 task suite.Source: Table 2 · p. 10

Metrics

  • Accuracy is computed by exact matching or CLAP-based retrieval, while verbose multiple-choice outputs are normalized with regex extraction.Source: §4 · p. 10

5. Results

AF-Next delivers broad gains across audio understanding, music, long-audio reasoning, speech recognition, voice understanding, and speech translation. Its variants provide additional benefits on challenging reasoning and captioning tasks.

Audio understanding and reasoning

  • AF-Next-Instruct substantially outperforms prior models across diverse audio understanding and reasoning benchmarks, with AF-Next-Think and AF-Next-Captioner adding further gains.The reported results span general audio, music, long-context audio, speech recognition, voice understanding, and translation.Source: §5 · pp. 10–11
  • 75.76 average accuracy is achieved on MMAU-v05.15.25 by AF-Next-Captioner, exceeding Audio Flamingo 3’s 72.42.Gains extend across sound, music, and speech subcategories.Source: §5 · p. 10

Music understanding

  • 8.8 coverage and 8.9 correctness are achieved on SongCaps, improving over AF3’s 6.7 and 6.2 for music captioning.AF-Next also reports strong music classification and instrument-recognition results.Source: §5 · p. 11

Long audio understanding

  • 73.9 on LongAudioBench and 81.2 on its +Speech variant exceed AF3 and Gemini 2.5 Pro, demonstrating strong long-context audio reasoning.The comparisons are 73.9 versus 68.6 and 60.4, and 81.2 versus 72.9 and 66.2, respectively.Source: §5 · p. 11

Automatic speech recognition

  • 1.54 WER on LibriSpeech test-clean and 2.76 on test-other establish new lows among LALMs, while AF-Next remains competitive across additional English ASR benchmarks.It also achieves the best WER on Common Voice 15, GigaSpeech, and VoxPopuli.Source: §5 · p. 11

Voice understanding and speech translation

  • 21.9 BLEU for EN→X Arabic translation improves over Phi-4-mm’s 9.9, while AF-Next leads on Chinese and remains competitive on Japanese and German.For X→EN translation, it also leads on Chinese and Arabic.Source: §5 · p. 11

6. Conclusion

AF-Next combines broad audio capabilities with stronger long-form reasoning and real-world robustness. The paper releases its code, checkpoints, and core techniques, including Temporal Audio Chain-of-Thought for timestamp-grounded reasoning.

Conclusion

  • AF-Next supports audio up to 30 minutes, multi-turn chat, timestamped captioning, and multilingual ASR while demonstrating stronger robustness in real-world use cases.These capabilities accompany state-of-the-art performance across contemporary audio understanding benchmarks.Source: §6 · p. 11
  • Temporal Audio Chain-of-Thought grounds intermediate evidence in time for long-audio question answering, enabling more faithful and robust reasoning.The paradigm explicitly links reasoning steps to timestamps.Source: §6 · p. 11
  • The authors open-source training code, model checkpoints, and core techniques to support future research in open audio-language modeling.The released model variants include AF-Next-Instruct, AF-Next-Think, and AF-Next-Captioner.Source: §6 · p. 11

Limitations

AF-Next’s limitations concern uneven internet-scale data coverage, difficult reasoning over long contexts, and evaluation that does not fully cover several supported capabilities.

Data coverage

  • Internet-scale audio remains noisy and unevenly distributed across domains, languages, and acoustic conditions.Low-resource languages, rare sound events, and specialized real-world domains are underrepresented.Source: Limitations · p. 11
  • Future work should improve the diversity, balance, and coverage of open audio datasets.Source: Limitations · p. 12

Long-context reasoning

  • Robust reasoning remains challenging when relevant evidence is temporally distant, sparse, or distributed across multiple audio segments.This limitation persists despite support for audio inputs up to 30 minutes.Source: Limitations · p. 12
  • Temporal Audio Chain-of-Thought improves temporal grounding, but stronger long-context memory, retrieval, and evidence aggregation remain needed.Source: Limitations · p. 12

Evaluation coverage

  • Evaluation focuses on established benchmarks and does not yet fully cover multi-talker ASR, speaker diarization, timestamped captioning, or voice-to-voice interaction.Broader evaluation protocols for these capabilities are identified as an important next step.Source: Limitations · p. 12

B. Potential Risks

AF-Next inherits risks associated with large language models and may reflect biases in its internet-scale audio and text training distribution.

Model risks

  • AF-Next may generate toxic, biased, or hallucinated outputs, including factually incorrect responses for some long-audio inputs.The paper notes varying recognition accuracy across accents and dialects and difficulty aggregating evidence over extended temporal spans.Source: B. Potential Risks · p. 17

Mitigation

  • 386K safety and instruction-following fine-tuning samples teach refusal and abstention behavior to mitigate these risks.Source: B. Potential Risks · p. 17

C. Data Anonymization and Privacy Safeguards

The paper derives training data from public academic datasets and openly accessible internet audio while applying safeguards against linking samples to speaker identities.

Data sources

  • The training data comes from publicly available academic datasets and openly accessible internet audio.The paper states that it does not intentionally collect data naming or uniquely identifying individuals.Source: C. Data Anonymization and Privacy… · p. 17

Privacy safeguards

D. AF-Next Training Details

AF-Next training is described across multiple stages, with detailed configurations provided in Table 3.

Training settings

E. AF-Next Training Dataset Details

AF-Next’s training data are organized across pre-training and fine-tuning datasets, with composition tracked by dataset size, audio-QA pairs, and training epochs. The data-generation examples cover long-audio question answering, captioning, temporal reasoning, multi-speaker recognition, multi-turn chat, and safety responses.

Dataset composition

Long-audio data

  • Long-audio data generation targets multi-turn chat, counting, detailed captioning, needle-in-the-haystack retrieval, subscene captioning, and temporal understanding.These prompt families cover several complementary forms of long-context audio reasoning and description.Source: Figure 4 · p. 20, Figure 5 · p. 21, Figure 6 · p. 21

Specialized training data

  • Additional examples train timestamped captions, diarized multi-speaker transcripts, multi-turn audio chat, and safety-aligned refusal responses.The examples span music, speech, sports, documentaries, commentary, benign questions, and harmful requests.Source: Figure 10 · p. 25, Figure 11 · p. 26, Figure 12 · p. 26

Related papers

Loading 2604.10905v1…