PaperlayerView exact revision on arXiv

Source-linked AI summary

AudioLDM 2: Learning Holistic Audio Generation with Self-supervised Pretraining

Haohe Liu, Yi Yuan, Xubo Liu, Xinhao Mei, Qiuqiang Kong, Qiao Tian, Yuping Wang, Wenwu Wang, Yuxuan Wang, Mark D. Plumbley

arXiv:2308.05734v3cs.SDcs.AIcs.MMeess.ASeess.SP

TL;DR

Audio generation has developed specialized models for speech, music, and sound effects, but a unified approach that also produces intelligible speech remains a challenge. AudioLDM 2 introduces LOA from AudioMAE, uses GPT-2 for condition-to-LOA translation, and applies a self-supervised latent diffusion model for generation. Experiments report competitive or state-of-the-art performance across text-to-audio, text-to-music, and text-to-speech tasks.

  • Problem

    Audio generation is commonly divided into specialized subdomains, while a unified method for generating varied audio signals including intelligible speech remains unresolved.

  • Method

    AudioLDM 2 encodes audio into LOA with AudioMAE, translates conditioning information into LOA using GPT-2, and generates audio with a latent diffusion model pretrained self-supervised on unlabelled audio.

  • Results

    AudioLDM 2 achieves competitive or state-of-the-art performance on text-to-audio, text-to-music, and text-to-speech benchmarks.

  • Takeaways & Limitations

    LOA provides a universal audio representation that supports self-supervised latent-diffusion pretraining and a unified perspective on audio generation.

Abstract

from arXiv · show

Although audio generation shares commonalities across different types of audio, such as speech, music, and sound effects, designing models for each type requires careful consideration of specific objectives and biases that can significantly differ from those of other types. To bring us closer to a unified perspective of audio generation, this paper proposes a framework that utilizes the same learning method for speech, music, and sound effect generation. Our framework introduces a general representation of audio, called "language of audio" (LOA). Any audio can be translated into LOA based on AudioMAE, a self-supervised pre-trained representation learning model. In the generation process, we translate any modalities into LOA by using a GPT-2 model, and we perform self-supervised audio generation learning with a latent diffusion model conditioned on LOA. The proposed framework naturally brings advantages such as in-context learning abilities and reusable self-supervised pretrained AudioMAE and latent diffusion models. Experiments on the major benchmarks of text-to-audio, text-to-music, and text-to-speech demonstrate state-of-the-art or competitive performance against previous approaches. Our code, pretrained model, and demo are available at https://audioldm.github.io/audioldm2.

I. INTRODUCTION

AudioLDM 2 addresses the challenge of building separate, domain-specific audio generators by proposing a unified framework for audio, music, and intelligible speech. It represents audio with LOA, translates conditions into LOA with GPT-2, and synthesizes audio with a self-supervised latent diffusion model.

  • Motivation: Separate audio-generation subdomains commonly use task-specific inductive biases, leaving unified generation across audio types unresolved.Speech models use pitch and duration predictors, while music models use MIDI representations or domain-specific pretrained modules.
  • Proposed framework: AudioLDM 2 introduces LOA, a universal audio representation intended to capture both fine-grained acoustic and coarse-grained semantic information.LOA is derived from features extracted by AudioMAE, which is pretrained on diverse audio using generative and reconstructive objectives.
  • Proposed framework: GPT-2 translates conditioning information into AudioMAE features, while a latent diffusion model synthesizes audio conditioned on those features.The latent diffusion model can be pretrained self-supervised on large-scale unlabelled audio data.
  • Contributions: The framework supports conditional generation of audio, music, and intelligible speech without domain-specific inductive bias.This unified capability is the paper’s central model contribution.
  • Results: AudioLDM 2 achieves competitive or state-of-the-art performance across text-to-audio, text-to-music, and text-to-speech benchmarks.The reported evaluation sets are AudioCaps, MusicCaps, and LJSpeech, respectively.

II. RELATED WORK

Related work spans conditional audio generation, speech and music synthesis, and diffusion-based modeling. AudioLDM 2 positions itself as a unified framework across these subdomains while addressing unintelligible speech in prior text-to-audio systems.

  • Conditional audio generation: General audio-generation systems include autoregressive and latent-diffusion approaches, alongside image-to-audio, video-to-audio, and unconditioned audio language modeling.Examples include AudioGen, AudioLDM, Make-an-Audio, Im2Wav, SpecVQGAN, and AudioLM.
  • Conditional audio generation: Previous text-to-audio systems often generate unintelligible speech, despite progress in general audio generation.This limitation distinguishes general audio-generation evaluation from intelligible speech synthesis.
  • Speech and music generation: Text-to-speech generates speech from transcriptions, whereas text-to-music creates music clips from textual descriptions.Specialized TTS systems such as FastSpeech2, GradTTS, and NaturalSpeech have achieved high speech quality.
  • Unified audio generation: AudioLDM 2 proposes a unified framework encompassing speech, sound effects, and music generation.Its broader framework connects the subdomains covered separately in prior work.
  • Diffusion models: Diffusion models provide high sample quality but can be slow because generation is iterative in a high-dimensional data space.They have been applied to mel-spectrogram and waveform generation for speech and audio.

III. AUDIOLDM 2

AudioLDM 2 uses LOA as an intermediate representation connecting conditioning information to audio generation. GPT-2 translates conditions into LOA, while a latent diffusion model generates audio from LOA and can be trained self-supervised.

  • III. AUDIOLDM 2: LOA bridges conditioning information and audio by representing audio semantics in an intermediate feature that is easier to model than the waveform.The framework defines Y = A(x) as the LOA encoding and uses it between condition C and audio x.
  • III. AUDIOLDM 2: AudioLDM 2 first translates conditions from modalities such as audio or text into estimated LOA, then generates audio conditioned on that estimate.The two stages are M: C → Ŷ and G: Ŷ → x.
  • III. AUDIOLDM 2: During training, the latent diffusion generator uses ground-truth LOA derived from available audio, enabling self-supervised optimization without audio annotations.The conditioning LOA is Y = A(x) rather than the estimated Ŷ = M(C).
  • III. AUDIOLDM 2: The framework uses GPT-2 for autoregressive LOA modeling and a latent diffusion model for waveform generation, with ground-truth or GPT-2-generated LOA used in training contexts.GPT-2-generated LOA is used when the ground-truth LOA is unavailable during inference.

B. Audio Representation Learning

AudioLDM 2 adopts AudioMAE as a general semantic representation learned from unlabeled audio and combines it with VAE-based reconstruction. The AudioMAE pipeline converts mel-spectrogram patches into embedded features, with reconstruction effects assessed across pooling choices.

  • B. Audio Representation Learning: AudioLDM 2 uses AudioMAE for general audio representation because it is self-supervised, pretrained on diverse audio, and suitable for speech, music, and sound effects.The representation is selected for generality and downstream audio-classification accuracy.
  • B. Audio Representation Learning: AudioMAE learns from unlabeled audio by masking mel-spectrogram patches and reconstructing the masked patches with encoder and decoder networks.This generative pretraining objective distinguishes it from discriminative approaches such as wav2vec and BYOL-A.
  • B. Audio Representation Learning: AudioMAE has been verified in the general audio domain, whereas representative alternatives such as MERT and HuBERT focus on music and speech, respectively.The passage cites AudioSet pretraining and downstream audio-classification performance for AudioMAE.
  • B. Audio Representation Learning: The AudioMAE encoder processes a log mel spectrogram by splitting it into P × P patches and embedding them into features with dimension D.The resulting feature shape is T′ × F′ × D, with T′ = T/P and F′ = F/P.
  • B. Audio Representation Learning: When λ = 1, reconstruction from the pooled LOA closely resembles the ground truth, while λ = 2 or 4 produces slight divergence but preserves semantic content.The comparison concerns audio reconstruction using the latent diffusion model.

2) AudioMAE Feature Post Processing:

AudioMAE features are pooled into shorter LOA embedding sequences for easier autoregressive estimation, while VAE features provide compressed audio representations with strong reconstruction ability. The section also describes GPT-2-based LOA prediction from flexible conditioning inputs.

  • 2) AudioMAE Feature Post Processing:: Two-dimensional average-max pooling converts AudioMAE features E into LOA Yλ, reducing sequence length while preserving time-frequency relationships.The pooling kernel and stride are both λ, and the output has shape Lλ × D.
  • 2) AudioMAE Feature Post Processing:: The pooled LOA sequence length is Lλ = T′F′/λ2, with λ selected so that Lλ remains a positive integer.Larger λ produces a shorter sequence for the subsequent conditioning-to-LOA model.
  • 3) Acoustic Representation Learning with VAE:: The VAE compresses mel-spectrograms into a lower-dimensional representation z and reconstructs them as X̂, with HiFiGAN converting X̂ into waveform x̂.VAE optimization uses reconstruction, discriminative, and KL-divergence losses.
  • 3) Acoustic Representation Learning with VAE:: AudioMAE organizes similar ESC50 audio clips more semantically than the VAE feature space in a tSNE visualization of ten randomly selected classes.Each plotted point represents an audio clip.
  • C. Conditioning Information to LOA Translation with GPT-2: GPT-2 predicts continuous LOA vectors autoregressively from conditioning information that may include audio, text, phonemes, or visual clues.The model is trained with mean squared error and does not discretize the AudioMAE feature space.
  • C. Conditioning Information to LOA Translation with GPT-2: CLAP supplies default text or audio conditioning, while FLAN-T5 adds textual semantic and temporal information that CLAP may not capture.The CLAP audio encoder is used for text-to-speech settings without captions.

D. LOA to Audio Generation with Latent Diffusion Model

AudioLDM 2 generates audio from LOA using a latent diffusion model in a compressed VAE space. The model learns reverse diffusion conditioned on LOA and uses a Transformer-UNet architecture for denoising.

  • Latent diffusion process: The latent diffusion model learns reverse diffusion from a Gaussian prior to the audio distribution in a compressed VAE latent space.This reduces computational cost compared with modeling the training data directly.
  • Latent diffusion process: The forward process progressively blends noise into VAE representations through a parameter-free T-step Markov process.The noise schedule β_t controls how quickly noise is added, producing a near-standard Gaussian at the final step.
  • Latent diffusion process: LOA serves as the condition signal for the reverse diffusion process, whose objective is derived using the evidence lower bound and KL divergence.The target conditional diffusion distribution has a closed-form solution given the original and noisy latent representations.
  • Architecture: A Transformer-UNet performs denoising with encoder downsampling, decoder upsampling, same-scale skip connections, and inserted transformer layers.The architecture is similar to AudioLDM’s UNet but contains more transformer layers.
  • Guidance illustration: Figure 4 illustrates samples generated for “A cat is meowing” under different classifier-free guidance scales.The figure varies the guidance scale while keeping the text prompt fixed.

2) Classifier-free Guidance:

Classifier-free guidance controls diffusion sampling by training conditional and unconditional models and combining their noise estimates with a guidance scale.

  • Classifier-free guidance: During training, the condition Y is randomly discarded with a fixed probability to learn both conditional and unconditional latent diffusion models.The paper gives 10% as an example discard probability.
  • Classifier-free guidance: During generation, LOA or an estimated LOA conditions sampling through a modified noise-estimation function.The modified estimator combines conditional and unconditional predictions at each sampling step.
  • Classifier-free guidance: The parameter w determines the classifier-free guidance scale used during sampling.Changing w changes the balance between unconditional and conditional noise estimates.

3) Joint Finetuning:

The experiments evaluate AudioLDM 2 across broad audio datasets and multiple generation tasks using objective, embedding-based, and human-rated metrics.

  • Datasets: The study uses AudioSet, WavCaps, AudioCaps, VGGSound, Free Music Archive, Million Song Dataset, LJSpeech, and GigaSpeech.AudioSet contains around two million ten-second audio clips across 527 classes, while WavCaps contains 403,050 clips.
  • Evaluation metrics: Text-to-audio evaluation combines FAD, KL divergence, Overall Impression, Audio and Text Relation, and CLAP score.These metrics assess audio quality, label-distribution similarity, perceived quality, text relevance, and audio-text correspondence.
  • Evaluation metrics: CLAP score measures correspondence between generated audio and the text prompt, with values ranging from −1 to 1 and larger values indicating stronger correlation.The score is computed from CLAP audio- and text-encoder outputs using cosine similarity with a small stabilizing constant.
  • Evaluation metrics: Text-to-music uses a similar evaluation protocol, while text-to-speech is evaluated with mean opinion score.The paper therefore uses task-specific subjective evaluation alongside the text-to-audio metrics.

C. Subjective Evaluation

Subjective evaluation uses crowd-sourced ratings to assess perceived quality, text relevance, and speech quality, alongside experimental configurations for different model and conditioning choices.

  • Subjective metrics: Amazon Mechanical Turk raters score Overall Impression, Audio and Text Relation, and MOS using instructions with examples.OVL and REL use five-point scales for quality and text relevance, respectively.
  • Model configurations: The experiments compare AudioLDM 2 and AudioLDM 2-Large, using n_trans = 2 and n_trans = 6 transformer layers, respectively.Both model sizes use a pretrained AudioMAE encoder.
  • Model configurations: AudioMAE produces a 768-dimensional feature sequence of length 512 for every ten seconds of mel spectrogram using non-overlapping 16 × 16 patches.The resulting features are pooled to form LOA representations for subsequent processing.
  • Experimental setup: The primary experiments combine five datasets, while † models use CLAP text and FLAN-T5 conditioning and ‡ models use CLAP audio and phoneme conditioning.The conditioning choices distinguish general audio and music generation from speech-oriented generation.

E. Training and Inference Setup

AudioLDM 2 trains GPT-2 and a latent diffusion model separately before applying the system across audio, music, and speech datasets. On AudioCaps, AudioLDM 2-AC outperforms prior systems on objective metrics, while larger-scale training shows distribution-related limitations.

  • Training and Inference Setup: The latent diffusion and GPT-2 models are initially trained separately, with random λ values used to improve robustness to varying conditioning lengths.The latent diffusion model is trained on 10-second audio segments padded to 10.24 seconds.
  • Training and Inference Setup: Training uses eight NVIDIA A100 80GB GPUs, AdamW, a 10^-4 learning rate, and 10,000 linear warmup steps without decay.DDIM sampling uses a classifier-free guidance scale of 3.5.
  • Evaluation Setup: The evaluation covers text-to-audio, text-to-music, and text-to-speech using AudioCaps, MSD, and LJSpeech, with AudioLDM 2-Full trained on 29,510 hours.The systems are denoted AudioLDM 2-AC, AudioLDM 2-MSD, and AudioLDM 2-LJS.
  • Scaling Results: FAD improves with model size, but larger training data degrades objective metrics, likely because the test distribution is narrower than the training distribution.Full-scale training nevertheless achieves FAD 1.42–2.13 versus AudioLDM-M’s FAD 4.53.

B. Text-to-Music Generation

AudioLDM 2 is evaluated for text-to-music generation against several strong baselines on MusicCaps. The general AudioLDM 2-Full model performs strongly, while speech results remain competitive with FastSpeech2 and improve with diverse pretraining.

  • Text-to-Music Results: AudioLDM 2-Full outperforms MusicGen by 36%, 11%, and 3.4% on FAD, KL, and CLAP scores, respectively.The comparison is conducted on the MusicCaps evaluation set.
  • General Versus Specialized Training: The general AudioLDM 2-Full model outperforms the music-only AudioLDM 2-MSD on objective metrics, supporting general audio training for specialized music generation.AudioLDM 2-Full also achieves a REL score of 3.54.
  • Textual Understanding: AudioLDM 2-Full achieves a REL score of 3.54, indicating stronger textual understanding than the other compared systems.Some baseline metrics are unavailable, and AudioLDM-M’s CLAP score may not reflect true performance because it was directly conditioned on CLAP during training.
  • Text-to-Speech Results: Without GigaSpeech pretraining, AudioLDM 2-LJS achieves MOS 3.65, compared with 3.78 for FastSpeech2.GigaSpeech pretraining produces greater fluctuations in emotion, punctuation, and tone.

D. Ablation Studies

Ablation studies examine how joint training and conditioning choices affect AudioLDM 2 on AudioCaps. The results show trade-offs between audio quality and audio-text alignment across conditioning configurations.

  • Joint Finetuning: Disabling joint GPT-2 and latent-diffusion finetuning causes marked deterioration across all three evaluation metrics.The ablation indicates that joint finetuning helps the two models cooperate.
  • Conditioning Mechanisms: Removing T-UNet cross-attention improves FAD from 1.67 to 1.38 but degrades KL divergence and CLAP scores.AudioMAE conditioning alone improves FAD, while FLAN-T5 conditioning adds language semantic information for audio-text relationships.
  • Conditioning Choices: Directly predicting LOA from text without CLAP and FLAN-T5 representations retains FAD 2.11 and KL 1.06 but noticeably degrades CLAP.The results suggest these representations can improve the relationship between text and generated audio.
  • Conclusion: AudioLDM 2 achieves state-of-the-art or comparative performance across text-to-audio, text-to-music, and text-to-speech tasks.The conclusion identifies multi-task GPT-2 training and alternative audio self-supervised representations as future directions.
Loading 2308.05734v3…