PaperlayerView exact revision on arXiv

Source-linked AI summary

AudioLDM: Text-to-Audio Generation with Latent Diffusion Models

Haohe Liu, Zehua Chen, Yi Yuan, Xinhao Mei, Xubo Liu, Danilo Mandic, Wenwu Wang, Mark D. Plumbley

arXiv:2301.12503v3cs.SDcs.AIcs.MMeess.ASeess.SP

TL;DR

Text-to-audio generation has been limited by quality, computational cost, and scarce paired data. AudioLDM uses CLAP-conditioned latent diffusion and achieves state-of-the-art quality while enabling zero-shot text-guided audio manipulation.

  • Problem

    Previous text-to-audio systems face limited generation quality, high computational costs, and scarce, low-quality paired audio-text data.

  • Method

    AudioLDM learns mel-spectrogram representations in a VAE latent space with CLAP-conditioned latent diffusion, using audio embeddings for training and text embeddings for sampling.

  • Results

    AudioLDM achieves state-of-the-art TTA quality on subjective and objective metrics using AudioCaps and a single GPU, while enabling zero-shot text-guided audio manipulations.

  • Takeaways & Limitations

    AudioLDM offers a computationally efficient approach to high-quality text-to-audio generation and zero-shot audio manipulation.

  • Takeaways & Limitations

    The model’s sampling rate is insufficient, especially for music generation, and separately trained modules may be misaligned.

Abstract

from arXiv · show

Text-to-audio (TTA) system has recently gained attention for its ability to synthesize general audio based on text descriptions. However, previous studies in TTA have limited generation quality with high computational costs. In this study, we propose AudioLDM, a TTA system that is built on a latent space to learn the continuous audio representations from contrastive language-audio pretraining (CLAP) latents. The pretrained CLAP models enable us to train LDMs with audio embedding while providing text embedding as a condition during sampling. By learning the latent representations of audio signals and their compositions without modeling the cross-modal relationship, AudioLDM is advantageous in both generation quality and computational efficiency. Trained on AudioCaps with a single GPU, AudioLDM achieves state-of-the-art TTA performance measured by both objective and subjective metrics (e.g., frechet distance). Moreover, AudioLDM is the first TTA system that enables various text-guided audio manipulations (e.g., style transfer) in a zero-shot fashion. Our implementation and demos are available at https://audioldm.github.io.

1. Introduction

AudioLDM addresses the limited quality, data availability, and computational efficiency of text-to-audio generation with a continuous latent diffusion model conditioned by CLAP embeddings. It also supports zero-shot text-guided audio manipulation, including style transfer, super-resolution, and inpainting.

  • Motivation: Natural language offers broader conditioning than label-to-sound systems restricted to small label sets such as UrbanSound8K’s ten classes.The introduction motivates text-to-audio generation for personalized sound effects, music, or speech in augmented and virtual reality, games, and video editing.
  • Method: High-dimensional audio signals motivate learned latent or discrete representations for efficient modeling in text-to-audio systems.AudioLDM uses a continuous latent space learned by a mel-spectrogram-based VAE, whereas related systems such as DiffSound and AudioGen employ learned discrete representations.
  • Contribution: AudioLDM achieves high-quality text-to-audio generation with continuous latent diffusion models while maintaining computational efficiency.The model generates mel-spectrogram-based VAE latent representations and uses an LDM conditioned on contrastive language-audio embeddings.
  • Contribution: AudioLDM enables text-guided audio style transfer, super-resolution, and inpainting without task-specific fine-tuning.These manipulations are realized through the reverse diffusion process of a pretrained LDM.
  • Contribution: Using CLAP embeddings allows LDM training without language-audio pairs, while audio-only training yields a high-quality and computationally efficient TTA system.The approach is designed to better exploit low-quality or limited audio-text data, whose preprocessing can otherwise constrain performance.

2. Related Work

Prior text-to-audio systems learn discrete audio representations but require paired audio-text data, motivating strategies to address data scarcity and quality. Diffusion models offer strong synthesis quality, while latent-space diffusion reduces the computational burden of waveform modeling.

  • Text-to-Audio Generation: Text-to-audio systems by Yang et al. (2022) and Kreuk et al. (2022) learn discrete audio representations from language and decode them into waveforms, but require paired data.Both works propose methods addressing low quality and scarce audio-text pairs.
  • Text-to-Audio Generation: DiffSound (Yang et al., 2022) combines text encoding, VQ-VAE representation learning, decoding, and vocoding with mask-based text generation from audio labels to mitigate paired-data scarcity.Its example inserts masks around labels such as “dog bark” and “man speaking.”
  • Text-to-Audio Generation: AudioGen (Kreuk et al., 2022) uses a Transformer decoder to generate discrete waveform-compressed tokens, trains on 10 datasets, and augments data to improve training diversity.It preprocesses language descriptions into labels to better match class-label annotations and simplify language-audio pairing.
  • Diffusion Models: Diffusion models achieve state-of-the-art sample quality across image, speech, audio, and video synthesis, but iterative generation in high-dimensional spaces slows inference.Using diffusion in a smaller latent space reduces modeling complexity and inference costs, especially for redundant audio waveforms.

3. Text-Conditional Audio Generation

AudioLDM performs text-to-audio generation by modeling compressed mel-spectrogram representations with a latent diffusion model conditioned on pretrained CLAP text embeddings. Training uses audio-side CLAP representations, enabling text-free LDM training, audio-only mixup augmentation, and classifier-free guidance during sampling.

  • Latent diffusion formulation: AudioLDM estimates the conditional distribution of compressed mel-spectrogram latents z0 given CLAP text embeddings Ey, then denoises Gaussian noise to generate an audio prior.The latent z0 is derived from a compressed mel-spectrogram, while reverse diffusion predicts generation noise conditioned on Ey.
  • Latent diffusion formulation: The model learns audio priors from CLAP audio embeddings Ex but uses text embeddings Ey only during generation, avoiding text supervision during LDM training.This separates cross-modal representation learning from latent diffusion modeling while retaining text-conditioned sampling.
  • Audio augmentation: Audio-only mixup with λ sampled from Beta(5, 5) increases training pairs (z0, Ex) and improves robustness to CLAP embeddings without requiring corresponding text descriptions.Because language information is unnecessary during LDM training, audio pairs can be mixed independently of their original captions.
  • Classifier-free guidance: Classifier-free guidance trains conditional and unconditional diffusion models by randomly dropping Ex with a fixed probability, then applies a guidance scale during sampling.The method uses natural-language CLAP text embeddings Ey, allowing guidance to exploit detailed text descriptions for audio generation.
  • Variational autoencoder: A convolutional VAE encodes and decodes compressed mel-spectrogram representations, using reconstruction, adversarial, and Gaussian constraint losses before decoding generated audio priors.The decoder reconstructs the mel-spectrogram from the latent audio prior generated by the LDM.

4. Text-Guided Audio Manipulation

AudioLDM supports zero-shot text-guided style transfer, inpainting, and super-resolution by initializing reverse diffusion from noisy or masked latent representations. The manipulation strength is controlled by the diffusion timestep, while latent-space masks preserve observed audio and generate missing content conditioned on text.

  • Style Transfer: AudioLDM enables text-guided style transfer by starting reverse diffusion from a noisy latent representation of the source audio.The source audio is manipulated through a shallow reverse process conditioned on the text input.
  • Style Transfer: Larger manipulations occur at n0 = 3N/4, whereas n0 ≈ N discards source information and approaches text-to-audio generation.The timestep n0 controls how much source-audio information is retained.
  • Inpainting and Super-Resolution: Audio inpainting and super-resolution generate missing audio from observed portions while conditioning generation on a text prompt.Both tasks incorporate the observed part into the latent representation.
  • Inpainting and Super-Resolution: A latent-space observation mask is derived from observed mel-spectrogram time-frequency bins, leveraging the VAE’s convolutional structure to approximately preserve spatial correspondence.Observed bins are marked in latent space, separating retained observations from regions to generate.
  • Inpainting and Super-Resolution: Masked reverse diffusion retains the ground-truth observed latent content while generating the missing information with the text-to-audio model.The observed latent representation is noised using the forward process before the missing regions are synthesized.

5. Experiments

Experiments show that AudioLDM outperforms baseline text-to-audio systems across objective and subjective evaluations, with quality improving as model capacity and training-data scale increase. Additional studies identify audio-embedding conditioning, moderate compression, complex attention, and sufficient DDIM steps as important design choices, while demonstrating text-guided super-resolution and inpainting.

  • Main Results: AudioLDM-L-Full achieves the best AudioCaps quality, with an FD of 23.31, while AudioLDM-S already outperforms baseline models and AudioLDM-L further improves overall results.AudioLDM-S uses only AudioCaps, whereas AudioLDM-L-Full adds AudioSet, FreeSound, and BBC SFX; the AudioCaps comparison is reported in Table 1.
  • Main Results: AudioLDM substantially outperforms DiffSound in human evaluation, reaching approximately 64 for both OVL and REL versus 45.00 and 43.83, respectively.Larger models improve overall audio quality, and expanding the training data produces significant OVL and REL gains.
  • Main Results: On the AudioSet evaluation containing music, all three AudioLDM models outperform DiffSound by a large margin on every reported metric.The AudioSet evaluation is constructed from randomly selected AudioSet samples because AudioCaps excludes music-related audio.
  • Conditioning Information: Audio-only conditioning is favored over direct text conditioning because captions can be ambiguous, abstract, and incomplete, making text-audio training less stable.The conditioning comparison uses data augmentation and the AudioGen mixing strategy for fairness.
  • Compression Level: Compression level r=4 offers high quality with manageable computation; increasing compression degrades performance, yet r=16 remains on par with AudioGen on KL and better than DiffSound on all metrics.Using r=1 or r=2 is difficult on a single RTX 3090 and slows inference, so r=4 is the default.
  • Ablations and Sampling: Complex UNet attention outperforms one-layer self-attention, balanced sampling provides no improvement, and increasing DDIM steps improves generation quality.The experiments also evaluate guidance scale, finding w=3 best for FD and KL but not FAD; w=2 is used for comparison with prior work.

6. Conclusions … B. Latent Diffusion Model

AudioLDM combines CLAP representations with latent diffusion for efficient, high-quality text-to-audio generation and zero-shot audio manipulation. Its CLAP encoders learn joint text–audio embeddings, while its diffusion model conditions a StableDiffusion UNet on CLAP embeddings and time steps.

  • 6. Conclusions: AudioLDM achieves state-of-the-art text-to-audio quality with one training dataset, AudioCaps, and a single GPU, while supporting zero-shot text-guided audio style transfer and other manipulations.The method is presented as advantageous in generation quality, computational efficiency, and audio manipulation, with quality evaluated using subjective and objective metrics.
  • Appendix: The reported system links contrastive language-audio pretraining and latent diffusion to deliver computationally efficient text-to-audio generation and zero-shot audio manipulation.This summarizes the conclusions’ claimed advantages together with the two component designs described in the appendix.
  • A. Contrastive Language-Audio Pretraining: CLAP projects text and audio into a shared latent space to capture their cross-modal similarity, using LAION-Audio-630K, AudioSet with T5-augmented captions, AudioCaps, and Clotho for training.The pipeline follows Wu et al. (2022), and the listed datasets provide the training data for contrastive language-audio pretraining.
  • A. Contrastive Language-Audio Pretraining: The contrastive encoders represent each audio sample and text input with 512-dimensional embeddings produced by HTSAT-based audio and RoBERTa-based text encoders.These embeddings are trained using a symmetric cross-entropy objective with a learnable temperature parameter and batch size D.
  • B. Latent Diffusion Model: The latent diffusion model uses the StableDiffusion UNet backbone and conditions generation on both the diffusion time step and the CLAP embedding.The time step is mapped to a one-dimensional embedding and concatenated with the CLAP embedding as conditioning information.
  • B. Latent Diffusion Model: Because the conditioning vector is one-dimensional, AudioLDM replaces StableDiffusion’s cross-attention conditioning with direct conditioning on the concatenated time-step and CLAP features.This design choice is described as the conditioning mechanism for the AudioLDM UNet.

C. Variational Autoencoder · D. Vocoder

The system uses a convolutional VAE to compress and reconstruct mel-spectrograms in a continuous latent space, then employs HiFi-GAN to synthesize waveforms from reconstructed mel-spectrograms. Across compression levels, the VAE achieves reconstruction metrics comparable to the ground-truth-mel-plus-vocoder setting.

  • C. Variational Autoencoder: The convolutional VAE compresses mel-spectrograms into a small continuous latent space while preserving their time-frequency spatial correspondence.Both encoder and decoder use stacked convolutional modules built from ResNet blocks (Kong et al., 2021a), as visualized in Figure 7.
  • C. Variational Autoencoder: The decoder reconstructs mel-spectrograms from stochastic VAE encodings formed from learned means, variances, and Gaussian noise.During generation, the decoder receives the stochastic encoding ẑ = ẑµ + ẑσ · ϵ, with ϵ ∼ N(0, I).
  • C. Variational Autoencoder: VAE training combines mel-spectrogram reconstruction, adversarial, and Gaussian constraint losses.The reconstruction loss uses mean absolute error, while PatchGAN adversarial losses improve reconstruction quality by distinguishing real and fake patches (Isola et al., 2017).
  • C. Variational Autoencoder: The VAE is trained at compression levels r=4, 8, 16 with latent channels C = 8, 16, 32, respectively, using at least 1.5M steps on one NVIDIA RTX 3090 GPU.Training uses Adam with a learning rate of 4.5 × 10^-6 and batch size six on AudioSet, AudioCaps, Freesound, and BBC SFX.
  • C. Variational Autoencoder: Comparable metrics across r=4, 8, 16 show reliable VAE mel-spectrogram encoding and decoding against the GT Mel + Vocoder setting.Table 8 reports reconstruction performance for all three compression levels on the AudioSet evaluation set.
  • D. Vocoder: HiFi-GAN (Kong et al., 2020a) serves as the vocoder, using multi-period and multi-scale discriminators to enhance perceptual waveform quality.It is trained on AudioSet using 64-band mel-spectrograms extracted from 16,000Hz audio and default HiFi-GAN V1 settings.
  • D. Vocoder: The vocoder uses a learning rate initialized at 2 × 10^-4 with 0.999 decay, batch size 96, and six NVIDIA 3090 GPUs.The pretrained vocoder is released in the open-source implementation.

E. Experiment Details · F. The Effect of Finetuning · G. Computation Efficiency Comparison

The experiments use millions of ten-second audio segments and specified single-GPU training configurations, while fine-tuning improves AudioCaps evaluation metrics but may narrow distributional breadth. AudioLDM-S also demonstrates faster generation than DiffSound across batch sizes.

  • E. Experiment Details: Training uses 3,302,553 ten-second audio samples, with long FreeSound and BBC SFX recordings truncated to thirty seconds and segmented.AudioSet and AudioCaps samples are already ten seconds long.
  • E. Experiment Details: AudioLDM-S and AudioLDM-L train for 0.6M steps on one RTX 3090, while AudioLDM-L-Full trains for 1.5M steps on one A100.The default compression level is r=4; batch sizes are 5 or 8, and learning rates are 3 × 10^-5 or 10^-5 depending on the model.
  • E. Experiment Details: AudioLDM-L-Full is further fine-tuned on AudioCaps for 0.25M steps before evaluation.This additional fine-tuning targets improved AudioCaps performance.
  • E. Experiment Details: Human evaluation uses 70 randomly selected samples spanning AudioCaps, AudioSet, and real recordings, requiring each model to generate 60 samples.The set contains 30 AudioCaps samples, 30 AudioSet samples, and 10 real recordings used as spam cases; outputs are anonymized.
  • F. The Effect of Finetuning: Fine-tuning improves various metrics on the AudioCaps evaluation set, but higher performance there may not indicate better overall performance.The evaluation distribution resembles AudioCaps’s training distribution, whereas broader audio distributions may perform worse on this limited evaluation set.
  • G. Computation Efficiency Comparison: AudioLDM-S generates eight ten-second audios in 20 seconds, whereas DiffSound requires more than 40 seconds across different batch sizes.This comparison is measured on a single A100 GPU.
  • G. Computation Efficiency Comparison: Without classifier-free guidance, AudioLDM-S generates eight ten-second audios within ten seconds.With classifier-free guidance, AudioLDM-Small achieves this using 150 DDIM steps.

H. Limitations · I. Demos

The study identifies technical limitations in AudioLDM’s sampling fidelity and separately trained modules, while warning that released models could enable harmful or misleading audio generation.

  • H. Limitations: AudioLDM’s sampling rate is insufficient, particularly for music generation, motivating future exploration of 32 kHz or 48 kHz sampling.Higher-fidelity sampling rates could improve generated-audio quality.
  • H. Limitations: Because AudioLDM’s modules are trained separately, their representations may be misaligned, and the VAE’s learned latent space may not be optimal.
  • H. Limitations: The released technology and models could be abused to generate fake audio effects that provide misleading information.
  • H. Limitations: Future work should restrict sensitive text content to help prevent harmful audio generation.

Audio Style Transfer · Audio Super-Resolution

AudioLDM-S enables zero-shot audio style transfer whose similarity to the source varies with the shallow reverse-process starting point, and zero-shot super-resolution that preserves low frequencies while generating missing high frequencies. These capabilities are demonstrated across multiple transfer pairs and four 8-to-16 kHz audio examples.

  • Audio Style Transfer: Zero-shot audio style transfer with AudioLDM-S can shift drum beats toward ambient music using a shallow reverse process with varied starting points n0.With smaller n0, generated samples remain more similar to the drum-beat source; Figure 9 illustrates the transfer.
  • Audio Style Transfer: Figure 10 demonstrates zero-shot audio style transfer from trumpet to children singing.
  • Audio Style Transfer: Figure 11 demonstrates zero-shot audio style transfer from sheep vocalization to narration, monologue.
  • Audio Super-Resolution: Zero-shot audio super-resolution with AudioLDM-S is shown for violin, a woman sneezing, baby crying, and female speech.The examples use 8 kHz input samples and compare 16 kHz generated samples with 16 kHz ground truth.
  • Audio Super-Resolution: AudioLDM-S retains the observed low-frequency content below 8 kHz while generating the missing 8-to-16 kHz high-frequency component.This behavior is visualized by comparing the processed high-resolution output with the ground-truth audio.
  • Audio Super-Resolution: Figure 12 presents the zero-shot audio super-resolution examples across the four audio categories.

Audio Inpainting · Environment Control

AudioLDM-S demonstrates zero-shot audio inpainting that preserves observed audio while controlling missing content through text prompts. It also controls the acoustic environment of generated speech samples by varying environment descriptions under a shared random seed.

  • Audio Inpainting: Four zero-shot inpainting examples remove the 2.5–7.5-second segment from 10-second ground-truth samples and regenerate it using the original text prompt.The figure compares the unprocessed input, generated inpainting result, and ground truth.
  • Audio Inpainting: The inpainting demonstrations show that AudioLDM-S can reconstruct missing audio content without requiring task-specific supervision.These are explicitly presented as zero-shot audio inpainting examples.
  • Audio Inpainting: AudioLDM-S preserves the observed signal while controlling inpainted content with text prompts such as ambient music, speech with bird calls, cat meowing, or rain and wind.The demonstration uses one observed signal and four different prompts, with the observed audio preserved in every generated sample.
  • Environment Control: With the same random seed and shared prompt “A man is speaking in,” AudioLDM varies the acoustic environment according to descriptions of a small room, huge room, huge room without background noise, or a studio.The examples demonstrate environment control through text while holding the common speech description and random seed fixed.
  • Environment Control: The environment-control examples indicate that AudioLDM can generate speech with distinct room and background-noise conditions specified by text.The compared conditions include room size, removal of background noise, and a studio setting.
  • Environment Control: Figure 15 presents examples of controlling acoustic environment with AudioLDM-S.The figure covers the four text-described environment variants generated from the shared speech prompt.

Music Control … Music Generation

AudioLDM-S demonstrates text-controlled generation across music attributes, pitch, materials, temporal order, general audio, novel prompts, and genre-specific music. The examples show that text descriptions can guide both compositional structure and acoustic characteristics.

  • Music Control: Text prompts control musical instruments, speed, and foreground-background relations in generated music.
  • Pitch Control: Text-controlled prompts generate sine waves with low, medium, or high pitch.
  • Material Control: AudioLDM-S controls generated audio materials through contrasting object and environment descriptions, including wooden and metal settings.
  • Temporal Order Control: When prompts contain multiple sound effects, AudioLDM preserves their described temporal order in the generated compositional audio.
  • Text-to-Audio Generation: AudioLDM-S generates examples spanning natural-environment sound effects, human speech, human activity, and object interactions.
  • Novel Audio Generation: AudioLDM-S generates audio from rarely seen descriptions, demonstrating generalization to novel prompts.
  • Music Generation: Using AudioSet labels as text descriptions, AudioLDM-S generates music with specified genres such as classical music.
Loading 2301.12503v3…