Source-linked AI summary
AudioGen: Textually Guided Audio Generation
Felix Kreuk, Gabriel Synnaeve, Adam Polyak, Uriel Singer, Alexandre Défossez, Jade Copet, Devi Parikh, Yaniv Taigman, Yossi Adi
TL;DR
Text-to-audio generation must handle difficult source separation, limited annotations, and long sequences required for high-fidelity audio. AudioGen combines a learned discrete audio representation with autoregressive modeling, mixing augmentation, multi-stream processing, and classifier-free guidance; it outperforms evaluated baselines on objective and subjective metrics and supports conditional and unconditional audio continuation.
Problem
Text-to-audio generation must address overlapping sound sources, reverberation and background noise, scarce text annotations, and extremely long sequences for high-fidelity audio.
Method
AudioGen uses an autoregressive model over learned discrete audio tokens, with text and audio mixing, multi-stream modeling, and classifier-free guidance.
Results
AudioGen outperforms evaluated text-to-audio baselines on objective and subjective metrics and supports conditional and unconditional audio continuation.
Takeaways & Limitations
AudioGen demonstrates that autoregressive models can generate high-quality audio conditionally or unconditionally from learned discrete representations.
Takeaways & Limitations
The approach can produce lower-quality audio with multi-stream relaxation, lacks temporal ordering of compositions, often generates unintelligible speech, and may reflect demographic or geographic dataset bias.
Abstract
from arXiv · showhide
We tackle the problem of generating audio samples conditioned on descriptive text captions. In this work, we propose AaudioGen, an auto-regressive generative model that generates audio samples conditioned on text inputs. AudioGen operates on a learnt discrete audio representation. The task of text-to-audio generation poses multiple challenges. Due to the way audio travels through a medium, differentiating ``objects'' can be a difficult task (e.g., separating multiple people simultaneously speaking). This is further complicated by real-world recording conditions (e.g., background noise, reverberation, etc.). Scarce text annotations impose another constraint, limiting the ability to scale models. Finally, modeling high-fidelity audio requires encoding audio at high sampling rate, leading to extremely long sequences. To alleviate the aforementioned challenges we propose an augmentation technique that mixes different audio samples, driving the model to internally learn to separate multiple sources. We curated 10 datasets containing different types of audio and text annotations to handle the scarcity of text-audio data points. For faster inference, we explore the use of multi-stream modeling, allowing the use of shorter sequences while maintaining a similar bitrate and perceptual quality. We apply classifier-free guidance to improve adherence to text. Comparing to the evaluated baselines, AudioGen outperforms over both objective and subjective metrics. Finally, we explore the ability of the proposed method to generate audio continuation conditionally and unconditionally. Samples: https://felixkreuk.github.io/audiogen
1 INTRODUCTION
AudioGen addresses text-conditioned audio generation, where overlapping sound sources, recording conditions, scarce annotations, and long high-fidelity sequences make controllable synthesis difficult. It introduces an autoregressive model with a learned discrete audio representation and reports stronger objective and subjective performance than prior work.
- Motivation: AudioGen targets high-fidelity, controllable, and diverse audio generation from descriptive text captions.The intended applications include soundscapes, music, speech, movies, video games, and virtual environments.
- Challenges: Audio is difficult to generate compositionally because its one-dimensional structure and reverberation hinder separating overlapping acoustic objects.Complex prompts may require multiple foreground and background categories with different durations and temporal positions.
- Approach: AudioGen is an autoregressive model whose first stage compresses raw audio into a compact discrete token sequence designed to preserve fidelity.A neural audio compression model reconstructs the input audio and uses perceptual discriminators during training.
- Results: AudioGen outperforms existing text-to-audio work on objective and subjective metrics and extends to conditional and unconditional audio continuation.The paper specifically reports more natural-sounding unseen audio compositions.
- Approach: The method improves text adherence with classifier-free guidance and compositionality through on-the-fly text and audio mixing.It also explores 285M- and 1B-parameter model variants and multi-stream transformers to trade off fidelity and sampling time.
2 RELATED WORK
Prior work established discrete audio representations and text-to-audio generation using diffusion models, while AudioGen develops a distinct autoregressive approach. Its differences include raw-waveform representation learning, compositional augmentation, classifier-free guidance, and evidence that autoregressive models can generate high-quality text-conditioned audio.
- Speech Representation Learning: Speech representation learning includes reconstruction and self-supervised methods that impose constraints such as temporal smoothness, discreteness, or hierarchy.These representations have supported speech recognition, phoneme segmentation, and audio compression.
- Discrete Audio Representations: Recent audio-generation work models discrete representations using quantized self-supervised features, language modeling, and multi-stream processing.Related applications include speech emotion conversion, spoken dialogue, and speech-to-speech translation.
- Text-to-Image: Text-to-image systems commonly tokenize visual content and model text-image codes autoregressively, while later approaches use diffusion and cascaded upsampling.These systems provide related precedents for conditioning generative models on text.
- Text-to-Audio: DiffSound generates audio from text with diffusion over VQ-VAE codes learned from mel-spectrograms and found its autoregressive Transformer inferior to diffusion.It also pre-trains with labeled tags and random input masking.
- Text-to-Audio: AudioGen differs from DiffSound by learning representations from raw waveforms, mixing text and audio for complex compositions, and applying classifier-free guidance autoregressively.The authors also report high-quality text-conditioned audio from the autoregressive setting.
3 METHOD
AUDIOGEN uses a discrete audio representation and an autoregressive Transformer language model conditioned on text to generate audio tokens that are decoded into waveforms. The method adds adversarial and perceptual reconstruction training, classifier-free guidance, and multi-stream modeling to improve fidelity, text adherence, and sequence efficiency.
- Audio representation: AUDIOGEN first encodes raw audio into discrete tokens with an encoder, vector-quantization layer, and decoder trained for waveform reconstruction.The representation uses a single codebook containing 2048 codes, each a 128-dimensional vector.
- Audio language modeling: The audio language model concatenates text and audio embeddings, then trains a Transformer decoder with causal self-attention and cross-attention to text.Text features come from a pretrained T5 encoder, while audio tokens are embedded through a look-up table.
- Audio representation: The audio representation is optimized with time- and frequency-domain reconstruction losses plus multi-scale discriminator and feature-matching losses.The generator combines weighted time, frequency, adversarial, and feature-matching objectives.
- Audio language modeling: Classifier-free guidance trains the Transformer conditionally and unconditionally, omitting text conditioning in 10% of training samples before combining probabilities at inference.The guidance scale γ controls the sampling combination.
- Audio language modeling: Multi-stream modeling shortens sequences by representing a length Tv sequence with k parallel streams of length Tv/k while maintaining approximately the same bitrate.Each stream uses a codebook of size 2048/k, and independently predicted stream embeddings are averaged at each time step.
4 EXPERIMENTS
The experiments evaluate AudioGen across dataset preparation, augmentation, objective and subjective metrics, baseline comparisons, audio continuation, and guidance-scale effects. Results show strong performance against DiffSound, benefits from mixing and classifier-free guidance, and a speed–quality trade-off in multi-stream models.
- Dataset: The study uses ten audio datasets with either multi-label annotations or natural-language captions, leaving approximately 4k hours after preprocessing and speech filtering.Multi-label tags are converted into pseudo-sentences, while stop words and numbers are removed and remaining words lemmatized.
- Data Augmentations: Audio mixing augmentation creates new compositions by fusing paired audio samples and concatenating their captions, with a randomly sampled SNR between −5 and 5.The augmentation is intended to support complex text-conditioned audio compositions.
- Evaluation Methods: The evaluation combines objective FAD and KL metrics with subjective overall-quality and text-relevance ratings.FAD reflects audio quality and correlates with human perception, while KL reflects broader audio concepts; the metrics are complementary.
- Main Results: AUDIOGEN-base outperforms DiffSound on all reported metrics while using fewer parameters, and AUDIOGEN-large significantly outperforms both systems.On the matched dataset setup, AUDIOGEN-base reaches KL 2.46 versus 2.57 and FAD 4.39 versus 7.39 for DiffSound.
- Audio Continuation: Audio continuation remains text-steerable with long prompts, although shorter audio prompts leave more room for textual guidance.Text and audio prompts have roughly the same impact at approximately 1.5s, while a gap between text-conditioned and unconditioned generations remains at approximately 4s.
- The effect of classifier-free guidance scale: Classifier-free guidance improves performance over vanilla sampling, with FAD minimized at γ = 3.0 and KL decreasing and converging at γ = 4.0.The authors identify γ = 3.0 as the best trade-off between quality and diversity in the evaluated setting.
- Multi-stream Results: Multi-stream representations offer faster inference but degrade generation performance relative to single-stream models, while learned representations remain comparable in ViSQOL.The multi-stream setting trades inference quality for speed, and its effective bitrate degradation contributes to lower SI-SNR values.
5 LIMITATIONS
AudioGen’s small down-sampling factor produces extremely long audio-token sequences, limiting long-range modeling and increasing inference time. The method also remains limited in temporal ordering, speech intelligibility, and dataset diversity.
- Sequence length: Small down-sampling factors create extremely long audio-token sequences, limiting long-range modeling and increasing inference time.The quality trade-off worsens for higher-resolution audio, such as 48kHz samples.
- Composition: Mixing augmentation improves source separation and complex compositions but does not capture temporal ordering between foreground and background events.The limitation includes distinguishing sequences such as barking followed by humming from simultaneous background humming.
- Speech: Omitting most speech samples often produces unintelligible speech, while additional speech data, augmentation, or speech features may mitigate this problem.
- Data diversity: The mainly YouTube-collected datasets underrepresent some demographic and geographic locations, potentially biasing generated samples.
6 CONCLUSION
AudioGen is a Transformer-based generative model operating on learned discrete audio representations. The work shows conditional and unconditional high-quality generation, while mixing augmentation improves performance and classifier-free guidance and multi-stream processing are evaluated.
- Conclusion: AudioGen operates on learned discrete audio representations with a Transformer-based generative model.
- Conclusion: Autoregressive models can generate high-quality audio samples both conditionally and unconditionally.
- Conclusion: On-the-fly text and audio mixing augmentations improve model performance, with ablations examining classifier-free guidance and multi-stream processing.
- Broader impacts: The work is positioned as a foundation for better text-to-audio models and future directions including benchmarking, semantic audio editing, and audio source separation.
7 APPENDIX
The appendix documents the datasets, continuation results, and model-size ablations. It includes qualitative evidence that text conditioning adds goat sounds during continuation and reports that the largest text encoder and audio language model combination performs best overall.
- Datasets: The dataset appendix lists ten audio datasets, all sampled at 16kHz.
- Additional results: Figure 4 compares unconditional and text-conditioned continuation using Mel-spectrograms for the prompt “speech and a goat bleating”.
- Additional results: With a speech-only audio prompt, unconditional continuation produces speech, whereas text-conditioned continuation generates both speech and goat sounds.
- Additional results: Table 4 reports full FAD and KL results across text-condition and audio-prompt settings.
- Model-size ablation: Using both T5-large and ALM-large yields the best overall KL and FAD results among the four model-size combinations.