PaperlayerView exact revision on arXiv

Source-linked AI summary

Audio Flamingo: A Novel Audio Language Model with Few-Shot Learning and Dialogue Abilities

Zhifeng Kong, Arushi Goel, Rohan Badlani, Wei Ping, Rafael Valle, Bryan Catanzaro

arXiv:2402.01831v3cs.SDcs.LGeess.AS

TL;DR

LLMs have limited understanding of non-speech sounds and non-verbal speech, motivating models that can reason over richer audio. Audio Flamingo combines audio modeling, ICL and retrieval, and dialogue-focused data strategies, achieving state-of-the-art results across multiple audio understanding and dialogue tasks. The paper also identifies larger language models, complex speech tasks, and broader multimodal integration as future directions.

  • Problem

    LLMs’ sound understanding remains largely limited to speech transcriptions, leaving non-speech sounds and non-verbal speech insufficiently addressed alongside few-shot adaptation and multi-turn dialogue.

  • Method

    Audio Flamingo combines a sliding-window audio extractor, cross-attention fusion, ICL and retrieval techniques, heterogeneous training data, and GPT-4-generated dialogue datasets.

  • Results

    Audio Flamingo achieves state-of-the-art results on multiple audio understanding tasks, few-shot benchmarks, and dialogue benchmarks without task-specific fine-tuning.

  • Takeaways & Limitations

    The model supports audio understanding with rapid adaptation to unseen tasks and multi-round user dialogue about audio.

  • Takeaways & Limitations

    Future work includes scaling to larger language models, handling complex speech-related tasks beyond transcription, and unifying audio with visual and other modalities.

Abstract

from arXiv · show

Augmenting large language models (LLMs) to understand audio -- including non-speech sounds and non-verbal speech -- is critically important for diverse real-world applications of LLMs. In this paper, we propose Audio Flamingo, a novel audio language model with 1) strong audio understanding abilities, 2) the ability to quickly adapt to unseen tasks via in-context learning and retrieval, and 3) strong multi-turn dialogue abilities. We introduce a series of training techniques, architecture design, and data strategies to enhance our model with these abilities. Extensive evaluations across various audio understanding tasks confirm the efficacy of our method, setting new state-of-the-art benchmarks. Our demo website is https://audioflamingo.github.io/ and the code is open-sourced at https://github.com/NVIDIA/audio-flamingo.

1. Introduction

Audio Flamingo addresses the limited sound understanding of LLMs by combining broad audio understanding, few-shot adaptation through ICL and retrieval, and multi-turn dialogue. Its architecture, training, data, and retrieval strategies yield strong benchmark and dialogue results without task-specific fine-tuning.

  • Audio Flamingo targets non-speech sounds and non-verbal speech, which remain outside LLMs’ primarily transcription-based sound understanding.
  • A sliding-window audio feature extractor captures temporal information, while cross attention fuses audio into the language model with linear rather than quadratic complexity in audio-token count.
  • Approximately 5.9 million audio-text pairs support a two-stage training approach using pre-training and supervised fine-tuning across heterogeneous audio tasks.
  • Efficient retrieval, an ICL template, interleaved ICL datasets, and a cross attention mask enable adaptation to new tasks without task-specific fine-tuning.
  • Two GPT-4-generated multi-turn dialogue datasets are used to fine-tune Audio Flamingo into a chat model with strong multi-turn dialogue ability.
  • The single model surpasses previous state-of-the-art results on most close-ended and open-ended benchmarks, while its chat version significantly outperforms dialogue baselines.

2. Related work

Related work covers multimodal LLMs, audio encoders, data augmentation, ICL, and RAG. Audio Flamingo is positioned as an audio understanding model combining broad sound coverage with ICL, retrieval, and multi-turn dialogue.

  • Multimodal LLMs: Prior multimodal LLMs process text alongside vision, audio, or multiple modalities using different integration methods.
  • Multimodal LLMs: Audio understanding in this paper includes non-speech sound, music, and non-verbal speech, extending beyond speech-focused systems.
  • Multimodal LLMs: Audio Flamingo is presented as the first audio understanding model with in-context few-shot learning, retrieval-augmented generation, and strong multi-turn dialogue abilities.
  • Audio encoders and representation: Audio encoders commonly derive spectrogram features with CNNs or Transformers, often targeting particular tasks such as speech recognition or event detection.
  • Data augmentation: Existing LLMs can generate captions or question-answering pairs from annotated tags, timestamps, and related information to augment audio data.
  • In-context learning: ICL adapts an LLM at inference time after it sees a few prompt examples, but general audio-understanding ICL has been much less explored.
  • Retrieval-augmented generation: RAG improves generation by using external knowledge, while prior audio retrieval work prepended similar-audio captions without providing the retrieved audio itself.

3. Method

Audio Flamingo combines sliding-window audio encoding, transformed audio representations, a decoder-only language model, and gated cross-attention. Its training uses weighted dataset losses, interleaved-sample masks, and staged optimization to support audio understanding and in-context learning.

  • 3.1. Architecture: Audio Flamingo uses a sliding-window audio extractor, trainable representation layers, a decoder-only language model, and gated xattn-dense layers.The architecture conditions the language model on audio representations through cross-attention and dense residual blocks.
  • 3.1. Architecture: Sliding windows preserve long-range and temporal information, with up to 16 windows supporting audio lengths of approximately 33.25 seconds.Each window spans 7 seconds with 5.25 seconds of overlap; longer audio is cropped and shorter audio is zero-padded.
  • 3.1. Architecture: The model takes interleaved audio and text as input and produces free-form text, while gated cross-attention layers condition each language-model block on audio.The gated layers combine cross-attention and dense residual blocks with tanh gating.
  • 3.2. Training Method: Training minimizes a weighted mixture of losses over non-interleaved and interleaved datasets using maximum likelihood estimation.Dataset weights are constant hyperparameters computed from predefined epoch counts.
  • 3.2. Training Method: For interleaved samples, block upper-triangular cross-attention masks condition each output on only the first j audio inputs.The interleaved loss is computed over all outputs, allowing examples to condition on varying numbers of prior multimodal inputs.
  • 3.2. Training Method: Training uses pre-training followed by supervised fine-tuning, initially training adaptation layers and later unfreezing the language model while keeping the audio encoder fixed.The staged procedure initializes the audio-conditioning layers before broader optimization.

4. Data

Audio Flamingo is trained on heterogeneous music, general-sound, and non-verbal-speech data for captioning, question answering, and classification. Retrieval-based interleaved examples, task templates, and generated dialogue data support few-shot adaptation and multi-turn interaction.

  • 4.1. Datasets: The training data cover music, non-speech general sound, and non-verbal speech across captioning, question-answering, and classification tasks.Classification targets include events, scenes, genres, instruments, and other sound labels.
  • 4.1. Datasets: ICL datasets retrieve top-k neighboring samples within each raw dataset using ℓ2 distance in fused LAION-CLAP audio-embedding space.Faiss-gpu accelerates the nearest-neighbor search, and retrieved samples provide interleaved audio, instructions, and outputs.
  • 4.1. Datasets: Dataset staging assigns lower-quality, less-diverse, or biased datasets to pre-training and uses different data during supervised fine-tuning.The selection considers data quality, diversity, source, and size, with weights computed from dataset epoch counts.
  • 4.1. Datasets: The dataset contains approximately 5.9 million audio-text pairs totaling approximately 18.1 thousand hours of audio.
  • 4.2. Templates: Templates encode task descriptions, audio markers, instructions, options, outputs, separators, and end-of-chunk boundaries for non-interleaved and ICL samples.Interleaved templates place retrieved similar examples before the target audio-instruction-output tuple.
  • 4.3. Dialogue Data: GPT-4-generated dialogue datasets based on AudioSet-SL and MusicCaps are filtered with LAION-CLAP thresholds to strengthen multi-turn dialogue training.The resulting datasets are named AF-Dialogue-AudioSetSL and AF-Dialogue-MusicCaps.

5. Experiments

Audio Flamingo is evaluated across in-distribution, zero-shot, few-shot, retrieval-augmented, and multi-turn dialogue benchmarks, including ablations of architecture and training data. It generally matches or surpasses strong baselines, while ICL-based retrieval improves unseen-task performance and larger training sets improve results on average.

  • Q1: Strong Audio Understanding Ability: Audio Flamingo matches or outperforms SOTA baselines on most in-distribution tasks, including benchmarks for captioning, question answering, and audio classification.The comparisons use task-specific reference values and evaluate one model across benchmarks, with a separate chat model for dialogues.
  • Q1: Strong Audio Understanding Ability: 10.4% higher accuracy on numerical question answering and 20.4% higher F1 on quality prediction demonstrate gains on specific audio-understanding tasks.On NSynth, source prediction accuracy is also 18.6% higher than baselines.
  • Q2: In-Context Few-Shot Learning: Audio Flamingo shows strong zero-shot generalization, performing better than SOTA methods on most evaluated zero-shot tasks.The zero-shot datasets are excluded from pre-training and supervised fine-tuning, including related parent and child datasets where applicable.
  • Q2: In-Context Few-Shot Learning: Over 10% average improvement for classification occurs with ICL-based RAG over zero-shot results, and Audio Flamingo significantly outperforms the SOTA retrieval-augmented captioning method on AudioCaps.The few-shot evaluation reports absolute improvements over the zero-shot results in Table 3.
  • Q3: Multi-Turn Dialogue: Audio Flamingo’s chat model is evaluated turn-by-turn on two generated dialogue datasets and significantly outperforms Qwen-Audio, LTU, and MU-LLaMA.Previous methods claimed multi-turn dialogue support, but the paper notes that quantitative evidence was unavailable for those claims.
  • Q4: Ablation Studies: Instruction-tuned OPT performs better in most tasks, while ClapCap favors open-ended tasks, Clap2023 favors close-ended tasks, and larger curated training sets improve results on average.Audio Flamingo also outperforms Pengi without extra data, and increasing pre-training and SFT data improves average benchmark results.

6. Conclusion and Future Work

Audio Flamingo combines strong audio understanding with few-shot adaptation, retrieval, and multi-turn dialogue capabilities, achieving state-of-the-art results across multiple audio benchmarks. The authors identify larger language models, complex speech tasks, and multimodal integration as future directions.

  • Audio Flamingo achieves state-of-the-art results on close-ended and open-ended audio understanding tasks without task-specific fine-tuning.
  • The model supports strong in-context learning, retrieval-augmented generation, and multi-turn dialogue, including state-of-the-art few-shot and dialogue benchmark results.
  • Future work includes scaling to larger language models and handling complex speech-related tasks beyond transcription through dense embeddings.
  • Using more ICL samples consistently improves few-shot results, although the benefit depends on the dataset.
  • Instruction-tuned language backbones outperform non-instruction-tuned backbones, while encoder performance varies between close-ended and open-ended tasks.
  • The authors also propose generating text and audio outputs, following more complex interleaved instructions, and combining audio understanding with visual language models.

Impact Statement

Audio Flamingo is intended to advance machine learning in the audio-language domain, with potential applications across several sectors. Its use requires attention to copyright compliance.

  • The proposed method may support automation in audio-language applications including education, healthcare, environment, industry, and music.
  • Careful use of the model is essential to ensure compliance with copyright restrictions.

A. Dataset Staging, Weights, and Templates

The dataset staging section organizes training data by task and provides instruction templates for audio understanding. It also documents the pre-training and supervised fine-tuning datasets and epoch counts.

  • Instructions are constructed for each training task and dataset, with templates covering sound or music description and audio question answering.
  • Additional templates support sound classification, bounded-length descriptions, speech-emotion recognition, and music instrument or genre identification.
  • Tables 7 and 8 report the datasets and epoch counts used for pre-training and supervised fine-tuning, respectively.

B. Generated dialogue datasets

The authors generate multi-turn audio dialogues from annotated sound and music datasets using GPT-4, then filter the outputs for quality and audio-text similarity. The resulting examples cover conversational reasoning about diverse audio events and instruments.

  • B. Generated dialogue datasets: AF-Dialogue-AudioSetSL is generated from strongly labeled AudioSet-SL annotations, while AF-Dialogue-MusicCaps is generated from MusicCaps tags and descriptions.
  • B. Generated dialogue datasets: GPT-4 prompts request multi-turn dialogues in which the assistant answers questions about audio events or music without including timestamps.
  • B. Generated dialogue datasets: The generation prompt requires later user questions to use pronouns for entities introduced in earlier turns and formats each question-answer pair as a JSON dictionary.
  • B. Generated dialogue datasets: The authors filter GPT-4 outputs containing unsupported phrases and remove QA samples whose LAION-CLAP audio-text cosine similarity is below 0.3.
  • B. Generated dialogue datasets: Generated dialogues include examples involving mechanical sounds, impacts, glass shattering, footsteps, and cowbell performance characteristics.

C.1. Additional Few-shot Experiments on Unseen Labels

Audio Flamingo adapts to classification tasks with labels unseen during training through ICL-based retrieval. It substantially improves over zero-shot performance, while outperforming Pengi even without extra data.

  • Audio Flamingo adapts to new classification tasks where none of the labels were seen during training.The demonstrated unseen labels are bird species and gun names.
  • The large improvement from zero-shot to few-shot indicates strong ability to follow retrieved in-context samples.
  • Table 9 reports Audio Flamingo accuracy on new classification tasks with unseen labels.
  • Audio Flamingo achieves better evaluation results than Pengi even without extra data.Larger-scale pretraining and supervised fine-tuning improve results on average.

C.3. Audio Flamingo understands sound in depth

Audio Flamingo demonstrates deeper audio understanding across speakers, temporal order, atmosphere, loudness changes, and spatial distance. Its outputs more closely match the described audio than the baseline examples across these dimensions.

  • Audio Flamingo correctly identifies three different voices when the baseline identifies only one.
  • Audio Flamingo preserves sound order by describing a man speaking followed by audience applause.
  • Audio Flamingo describes the music as having a melancholic and introspective atmosphere, unlike the baseline’s generic characterization.
  • Audio Flamingo recognizes a man speaking, a subsequent loud bang, and brief laughter-related context in a changing-loudness example.
  • Audio Flamingo identifies thunder in the background of speech and rain, capturing the sound’s distance cue.

C.4. Audio Flamingo looks at retrieved samples

Audio Flamingo uses retrieved samples selectively: it borrows relevant wording, ignores misleading examples, and supports detailed multi-turn audio dialogue. The examples also show recognition of overlapping events and vocal characteristics.

  • Retrieval-augmented captioning: Audio Flamingo borrows relevant keywords from retrieved samples to describe paper crumpling and crinkling.
  • Retrieval-augmented captioning: Audio Flamingo combines retrieved evidence to describe a woman speaking while faucet water runs.
  • Retrieval robustness: Audio Flamingo predicts pop and flute despite retrieved samples naming different music genres and instruments.
  • Dialogue understanding: Audio Flamingo describes bird, mechanical, impact, speech, and ticking sounds, whereas the baseline mentions only birds.
  • Dialogue understanding: Audio Flamingo correctly identifies that bird sounds are intermittent and overlap with the person’s speech.
  • Dialogue understanding: In dialogue examples, Audio Flamingo identifies male vocals and distinguishes R&B genre content from vocal descriptions.
Loading 2402.01831v3…