PaperlayerView exact revision on arXiv

Source-linked AI summary

Taming Data and Transformers for Audio Generation

Moayed Haji-Ali, Willi Menapace, Aliaksandr Siarohin, Guha Balakrishnan, Vicente Ordonez

arXiv:2406.19388v4cs.SDcs.CLcs.CVcs.MMeess.AS

TL;DR

Ambient audio generation is constrained by scarce, weakly annotated data, captioning limitations, and poor model-size scaling. The paper addresses these issues with a 47M-clip dataset pipeline, AutoCap, and scalable transformer-based GenAu, which improves captioning and generation metrics over baselines.

  • Problem

    Ambient audio generation lacks large, well-annotated audio-text datasets, accurate captions, and reliably scalable generation architectures.

    Source: §1 · pp. 1–2

  • Method

    The paper builds AutoReCap-XL through scalable filtering, uses metadata and visual cues with a Q-Former in AutoCap, and trains latent-diffusion GenAu with a scalable transformer architecture.

    Source: §1 · pp. 1–2, §3.1 · p. 4, §5 · p. 8

  • Results

    GenAu improves over comparable baselines by 4.7% in FAD, 11.1% in IS, and 13.5% in CLAP, while AutoCap improves CIDEr by 3.2%.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Jointly scaling ambient audio data, caption quality, and model architecture yields stronger reported captioning and audio-generation performance.

    Source: §1 · p. 2, §5 · p. 8

    AutoCap’s 4,892-word AudioCaps fine-tuning vocabulary limits caption diversity, and AutoReCap has only been validated for audio generation.

    Source: §5 · p. 9

Abstract

from arXiv · show

The scalability of ambient sound generators is hindered by data scarcity, insufficient caption quality, and limited scalability in model architecture. This work addresses these challenges by advancing both data and model scaling. First, we propose an efficient and scalable dataset collection pipeline tailored for ambient audio generation, resulting in AutoReCap-XL, the largest ambient audio-text dataset with over 47 million clips. To provide high-quality textual annotations, we propose AutoCap, a high-quality automatic audio captioning model. By adopting a Q-Former module and leveraging audio metadata, AutoCap substantially enhances caption quality, reaching a CIDEr score of $83.2$, a $3.2\%$ improvement over previous captioning models. Finally, we propose GenAu, a scalable transformer-based audio generation architecture that we scale up to 1.25B parameters. We demonstrate its benefits from data scaling with synthetic captions as well as model size scaling. When compared to baseline audio generators trained at similar size and data scale, GenAu obtains significant improvements of $4.7\%$ in FAD score, $11.1\%$ in IS, and $13.5\%$ in CLAP score. Our code, model checkpoints, and dataset are publicly available.

1. Introduction

Ambient audio generation is limited by scarce, weakly annotated data and uncertain model scaling. The work addresses both axes with AutoReCap-XL, AutoCap, and GenAu, reporting larger data, improved captioning, and stronger generation results.

Motivation

  • The work targets data and model scaling because high-quality text annotations and scalable generative models are key pillars of audio-generation quality.Ambient audio is treated as a distinct challenge alongside speech and music.Source: §1 · p. 1

AutoReCap-XL

  • 47 million ambient audio clips make AutoReCap-XL 75 times larger than previously available datasets through scalable transcription-based filtering.The approach avoids downloading audio or video and reduces reliance on expensive content classifiers.Source: §1 · pp. 1–2

Motivation

  • Ambient audio lacks extensive, well-annotated datasets because online-video collections are dominated by speech and music; AudioSet contains 99% speech or music clips.Prior filtering approaches used expensive classifiers and high filtering rates, limiting large-scale collection.Source: §1 · p. 1

AutoCap

  • 3.2% higher CIDEr on AudioCaps shows that AutoCap’s Q-Former, metadata, and visual cues improve automatic audio captioning.AutoCap supplies textual descriptions for AutoReCap-XL using metadata and visual information alongside audio representations.Source: §1 · p. 2

GenAu

  • 11.1% higher Inception Score, 4.7% better FAD, and 13.5% improvement in CLAP score over baselines demonstrate GenAu’s stronger audio generation and alignment.GenAu also improves consistently across metrics as model size increases.Source: §1 · p. 2

2. Related Work

Prior work uses transformer captioners, diffusion-based audio generators, and limited or noisy text-audio datasets. This work combines metadata- and visual-assisted captioning with a scalable transformer architecture and a 47M-clip collection pipeline.

Automatic Audio Captioning

  • AutoCap adds audio metadata and visual information to a lightweight Q-Former-based captioning model, extending encoder-decoder approaches for ambiguous sounds.The method uses an intermediate audio representation to connect the pretrained encoder and decoder.Source: §2 · p. 2

System Overview

  • Figure 1 contrasts AutoCap’s encoder-Q-Former-BART captioning pipeline with GenAu’s latent sequence attention architecture.The GenAu overview uses local, read/write, and global attention to communicate across latent groups.Source: Figure 1 · p. 3

Text-conditioned Audio Generation

  • GenAu proposes a transformer architecture with strong scalability properties for text-conditioned audio generation.It extends the field’s diffusion-based generation landscape with a transformer design.Source: §2 · pp. 2–3

Text-Audio Datasets

  • Existing human-captioned datasets contain only 51k audio-text pairs, while LAION-Audio has 630k samples with highly noisy raw descriptions.Prior video-based efforts used classifiers or weak captions that did not incorporate the audio signal itself.Source: §2 · p. 3
  • 47M audio clips form the largest available text-audio dataset through efficient extraction from video datasets and automatic captioning.The pipeline addresses the scarcity of high-quality large-scale ambient audio-text pairs.Source: §2 · p. 3

3. Method

The method combines AutoCap captioning, transcript-guided ambient-audio collection, and GenAu latent diffusion to scale text-to-audio generation. Its components address caption quality, dataset scarcity, and inefficient scaling in audio generation.

3.1. Automatic Audio Captioning

  • AutoCap connects pretrained audio encoders and BART through a Q-Former while incorporating video titles and captions as metadata.The Q-Former learns an intermediate representation, and the input sequence combines audio embeddings with metadata embeddings before BART predicts captions.Source: §3.1 · p. 4
  • A two-stage training procedure first aligns the Q-Former and projection layers on weakly labeled data, then adapts BART on AudioCaps while limiting knowledge drift.BART and the audio encoders are frozen during Stage 1; Stage 2 unfreezes BART except its embedding layer and uses a lower learning rate.Source: §3.1 · p. 4

3.2. Data Collection and Re-captioning Pipeline

  • Transcript gaps identify candidate audio segments without speech or music, which AutoCap captions before keyword and CLAP-similarity filtering.The pipeline removes captions containing speech or music keywords and filters audio-text pairs with CLAP similarity below 0.1.Source: §3.2 · pp. 4–5, Figure 2 · p. 4
  • AutoReCap-XL scales this pipeline to 47M ambient clips spanning 123.5k hours from 20.3M videos, forming the largest paired-caption dataset described.The broader AutoReCap dataset contains 761,113 audio-text pairs, while AutoReCap-XL adds clips collected from four large-scale video datasets.Source: §3.2 · p. 5

3.3. Scalable Text-2-Audio Generation

  • GenAu generates audio in the latent space of a 1D-VAE, replacing direct waveform modeling with lower-dimensional Mel-spectrogram latents.The 1D-VAE uses 1D convolutions because Mel-channel separation is nonlinear and poorly suited to common 2D autoencoder designs.Source: §3.3 · p. 5
  • GenAu adapts FIT into a latent-space transformer that uses local attention and a small set of global latent tokens to reduce computation.Read and write cross-attention transfers information between patch groups and latent tokens, enabling long-range interaction without full quadratic attention.Source: §3.3 · pp. 5–6
  • GenAu conditions diffusion on FLAN-T5 and CLAP text embeddings, the diffusion timestep, and dataset identity through cross-attention inside each FIT block.Dataset-ID conditioning adapts generation style across datasets, while training uses epsilon prediction with a linear noise scheduler.Source: §3.3 · p. 6

4. Experiments

The experiments evaluate AutoCap for audio captioning and GenAu for text-to-audio generation, including data, caption-quality, architecture, and model-size scaling. AutoCap outperforms captioning baselines, while GenAu improves generation metrics and user preference under comparable training scales.

4.1. Automatic Audio Captioning

  • 83.2 CIDEr and 73.1 BLEU1 are achieved by AutoCap, which outperforms baselines on all reported captioning metrics.Metadata substantially improves CIDEr, although it slightly reduces SPICE; AutoCap also generates captions four times faster than ENCLAP.Source: §4.1 · p. 7

4.2. Text-2-Audio Generation

  • 11.1% higher IS, 4.7% lower FAD, and 13.5% higher CLAPLAION are achieved by GenAu than similarly scaled state-of-the-art baselines.Using the full 10-second AutoReCap-XL subset further improves IS by over 21.8% and CLAP by 20.5%.Source: §4.2 · pp. 7–8
  • 56.3% higher IS is obtained when increasing synthetic training data, while AutoReCap-XL improves FD by 12.3% and CLAP by 11.6% over AutoReCap with matched evaluation settings.User studies consistently favor models trained with AutoReCap over models trained only on AudioCaps.Source: §4.2 · p. 8, Table 5 · p. 8
  • Higher-quality AutoCap captions improve GenAu across all metrics, whereas expanding data with lower-quality WavCaps captions provides no significant gain over AudioCaps alone.The result supports caption quality as a distinct contributor beyond dataset size.Source: §4.2 · p. 8
  • GenAu-L at 1.25B parameters outperforms GenAu-S at 493M across all metrics, including an almost 20.5% increase in IS and stronger user preference.This scaling trend contrasts with prior reports of diminishing returns for ambient-audio model-size scaling.Source: §4.2 · p. 8
  • FIT outperforms UNet and DiT backbones across all metrics, which the authors associate with more efficient computation on silent or redundant audio segments.The evaluation recomputes baselines with consistent metric protocols and includes an out-of-distribution split for ablations.Source: §4.2 · pp. 7–8

5. Conclusion

The conclusion presents a pipeline combining scalable data collection, AutoCap annotation, and a transformer-based latent diffusion generator. It also identifies limited caption vocabulary and incomplete validation beyond audio generation as boundaries of the work.

5. Conclusion

  • AutoCap annotates a 47M-clip dataset, which GenAu uses to improve large-scale ambient audio generation.The conclusion frames the work as a holistic response to captioned-audio data scarcity and generator quality.Source: §5 · pp. 8–9
  • AutoReCap-XL may support joint text-audio-video datasets and text-to-audio-video generation applications.The paper also suggests extending GenAu to speech and music.Source: §5 · p. 9
  • AutoCap was fine-tuned on AudioCaps with 4,892 unique words, limiting caption diversity and potentially making detailed-prompt generation challenging.AutoReCap has been validated only for audio generation, leaving broader task analysis for future work.Source: §5 · p. 9

A.1. Stage 1: Data Selection

AutoReCap-XL is constructed by selecting large video collections, extracting transcript-free audio, captioning segments, and filtering speech and music. The resulting dataset contains 47 million audio-text pairs and is reported as substantially larger and more accurately captioned than prior datasets.

A.1. Stage 1: Data Selection

  • 73 million videos from six video datasets are selected for likely strong audio-video correspondence before transcript-based processing.The sources include AudioSet, VGGSound, ACAV100M, VideoCC, YTTEMP1B, and HDVila-100M.Source: §A.1 · p. 16

A.2. Stage 2: Speech and Music Filtering

  • 327.3 million transcript-free segments from 55.1 million videos form AutoReCap-XL-Raw after retaining audio longer than one second.Videos without transcripts are discarded, and full videos are processed when source datasets provide only segments.Source: §A.2 · p. 16

A.3. Stage 3: Post-filtering of Speech and Music

  • AutoCap-captioned segments are limited to 10 seconds, and consecutive segments with identical captions are concatenated into longer clips.The resulting collection is summarized through caption-length, audio-duration, word-cloud, and CLAP-score distributions.Source: §A.2 · p. 16, Figure 9 · p. 17, Figure 11 · p. 17
  • LLM-generated speech and music keywords remove matching captions, followed by filtering audio-text pairs with CLAP scores below 0.1.The procedure addresses residual speech and music contamination after transcript-based selection.Source: §A.3 · p. 16
  • 47 million audio-text pairs from 20.3 million videos remain after post-filtering, making AutoReCap-XL 75 times larger than LAION-Audio-630K by clip count.The authors also report more accurate captions than datasets relying on raw textual data.Source: §A.3 · pp. 16, 18

B. Architecture details

AutoCap uses fine-grained HTSAT audio embeddings as its encoder representation. Frequency-averaged tokens are replicated and flattened to provide a 1024 × 768 sequence for the captioning architecture.

B. Architecture details

  • A 10-second, 32-kHz mono input becomes a mel-spectrogram with shape T = 1024 and F = 64 before HTSAT encoding.HTSAT uses a 1024 window, 320 hop size, and 64 mel bins.Source: §B.1 · p. 18
  • 32 × 32 × 768 fine-grained embeddings are flattened into 1024 × 768 after frequency averaging and token replication.The representation is adopted throughout the work, with additional evaluation reported for the replication operation.Source: §B.1 · p. 18

C. Limitations

The paper identifies limitations in AutoCap, GenAu, and AutoReCap-XL involving ambiguity, domain scope, vocabulary breadth, and insufficient downstream validation.

C.1. AutoCap

  • AutoCap struggles to disambiguate similar sounds, capture temporal relationships, separate foreground from background noise, and avoid repetitive captions.Its limited 4,892-word vocabulary contributes to repetitive outputs.Source: §C.1 · p. 18

C.2. GenAu

  • GenAu underperforms specialized models for music generation and text-to-speech synthesis and struggles with long, detailed prompts.The prompt limitation is linked to the restricted vocabulary of paired texts.Source: §C.2 · p. 18

C.3. AutoReCap-XL

  • AutoReCap-XL has 4,461 unique words and has not yet been extensively analyzed for caption accuracy or downstream-task performance.The constrained vocabulary is attributed to the AudioCaps-trained captioner.Source: §C.3 · p. 19

D. Evaluation Details

The evaluation details address checkpoint instability, metric inconsistency, and controlled user-study procedures across audio captioning and generation experiments.

D.1. Audio Captioning

  • Audio captioning results are reported at convergence after 20 pretraining and 20 fine-tuning epochs to reduce validation-checkpoint instability.The authors note weak validation–test correlation and substantial seed sensitivity in prior evaluation practice.Source: §D.1 · p. 19

D.2. Audio Generation

  • Baseline generation metrics were recomputed under identical configurations; GenAu outperforms baselines across metrics except FAD, where Make-An-Audio 2 is slightly better.A user study nevertheless indicates that GenAu is generally preferred over Make-An-Audio 2.Source: §D.2 · p. 19

D.3. User Study

  • Each user study used five participants and 1,000 responses, with 50 sampled clips drawn from the 200 longest-prompts in the AudioCaps test split.Seeds and other sampling parameters were fixed across experiments.Source: §D.3 · p. 19
  • Participants compared baseline clips for realism, quality, prompt faithfulness, and overall preference after hearing the generated audio and seeing its prompt.The interface screenshot is provided in Fig. 13, while the questions appear in Tab. 8.Source: §D.3 · p. 19

E. Training and Inference Details

The paper describes AutoCap and GenAu training configurations, contrasts GenAu with related architectures, and summarizes related captioning approaches and additional-results organization.

E.1. AutoCap

  • AutoCap adds 6.2M parameters, including a 4.7M Q-Former, 0.9M embedding layers, and 0.6M projection layers, to frozen HTSAT and BART models.Its Q-Former uses 256 learnable tokens, hidden dimension 256, eight attention heads, and two hidden layers.Source: §E.1 · p. 20
  • AutoCap uses Adam with learning rates of 10^-4 and 10^-5 across two stages and trains for nine hours on eight A100 80GB GPUs.Although trained on 10-second clips, it qualitatively generalizes to 1–2-second audio.Source: §E.1 · p. 20

E.2. GenAu

  • GenAu’s large model trains for 220k steps on 48 A100 80GB GPUs, while the small model trains for 210k steps on 32 GPUs.The models use LAMB, a 0.005 learning rate, cosine scheduling, 0.1 weight decay, and 0.1 dropout.Source: §E.2 · p. 20

F.1. Text-conditioned audio generation

  • GenAu uses FiT and is consistently preferred over a 937M-parameter DiT-based Make-An-Audio 2 baseline when trained at comparable data settings and smaller scale.Unlike Stable Audio Open’s VAE representation, GenAu uses Mel-spectrograms and can benefit from improved latent audio representations.Source: §F.1 · p. 21

F.2. Audio captioning

  • The paper trains dedicated AutoCap captions because reported zero-shot audio-language-model captioning results remain inferior to state-of-the-art automatic audio captioning methods.This choice supports caption quality for the AutoReCap dataset.Source: §F.2 · p. 21

G. Additional Results

  • The additional-results section provides supplementary evaluation and ablations for audio captioning and audio generation.The supplied passage introduces these results without reporting their outcomes.Source: G. Additional Results · p. 21

G.1. Additional Audio Captioning Evaluation

Additional experiments compare caption descriptiveness and ablate architectural and latent-space choices in audio generation.

G.1. Additional Audio Captioning Evaluation

  • AutoCap produces the most descriptive captions among the compared AAC methods, capturing the most elements from the ground-truth audio.ENCLAP and CoNeTTE tend to produce shorter captions.Source: §G.1 · pp. 21–22

G.2. Additional Audio Generation Evaluation

  • Removing either Flan-T5 or CLAP encodings reduces all metrics, while more latent tokens improve performance and patch size 2 decreases it.These findings evaluate fundamental architectural choices in the scalable FiT model.Source: §G.2 · p. 22
  • Larger 1D-VAE latent spaces lower reconstruction loss but hinder generation quality, motivating the use of 64 channels.The comparison reports effects on reconstruction loss, FAD, FD, and IS.Source: §G.2 · p. 22

G.3. Additional HTSAT Embedding Extraction Evaluation

The ablation evaluates token replication factors for HTSAT embeddings in HTSAT-BART and AutoCap on AudioCaps. Larger replication factors consistently improve performance, leading the authors to adopt 32x replication throughout their work.

G.3. Additional HTSAT Embedding Extraction Evaluation

  • The ablation compares 32, 256, and 1024 HTSAT tokens for HTSAT-BART, and 32 versus 1024 tokens for AutoCap.These correspond to no repetition, 8x repetition, and 32x repetition for the shared HTSAT procedure.Source: §G.3 · p. 22
  • 32x token replication consistently improves performance on the AudioCaps test split, so the authors adopt it for HTSAT embedding extraction.The reported explanation is that replication increases computation in the downstream model.Source: §G.3 · p. 22, Table 12 · p. 24, Figure 14 · p. 25
  • Evaluation follows the prior training procedure and uses the last obtained checkpoint on the AudioCaps test split.The results are reported in Table 12 and Figure 14.Source: §G.3 · p. 22

Related papers

Loading 2406.19388v4…