PaperlayerView exact revision on arXiv

Source-linked AI summary

Make-An-Audio: Text-To-Audio Generation with Prompt-Enhanced Diffusion Models

Rongjie Huang, Jiawei Huang, Dongchao Yang, Yi Ren, Luping Liu, Mingze Li, Zhenhui Ye, Jinglin Liu, Xiang Yin, Zhou Zhao

arXiv:2301.12661v1cs.SDcs.LGcs.MMeess.AS

TL;DR

Text-to-audio generation is constrained by scarce high-quality text-audio pairs and the difficulty of modeling long continuous audio. Make-An-Audio addresses these gaps with pseudo prompt enhancement, spectrogram autoencoding, CLAP, and diffusion, achieving state-of-the-art evaluations and multimodal X-to-Audio generalization.

  • Problem

    Text-to-audio generation lacks large-scale high-quality text-audio datasets and requires modeling complex long continuous waveforms.

    Source: §1 · p. 1, §2.2 · p. 2

  • Method

    Make-An-Audio combines distill-then-reprogram pseudo prompt enhancement, spectrogram autoencoding, CLAP representations, and latent diffusion for text-to-audio generation.

    Source: §1 · p. 2, §3.5 · p. 4

  • Results

    Make-An-Audio achieves new state-of-the-art text-to-audio results in objective and subjective evaluations, with realistic faithful synthesis and multimodal generalization.

    Source: §1 · p. 2, §7 · p. 8

  • Takeaways & Limitations

    The model generates high-definition, high-fidelity audio from user-defined text, audio, image, and video inputs.

    Source: §1 · p. 2

    Diffusion requires multiple iterative refinements, while latent diffusion typically requires more computational resources and can degrade with less training data.

    Source: H. Limitations · p. 16

Abstract

from arXiv · show

Large-scale multimodal generative modeling has created milestones in text-to-image and text-to-video generation. Its application to audio still lags behind for two main reasons: the lack of large-scale datasets with high-quality text-audio pairs, and the complexity of modeling long continuous audio data. In this work, we propose Make-An-Audio with a prompt-enhanced diffusion model that addresses these gaps by 1) introducing pseudo prompt enhancement with a distill-then-reprogram approach, it alleviates data scarcity with orders of magnitude concept compositions by using language-free audios; 2) leveraging spectrogram autoencoder to predict the self-supervised audio representation instead of waveforms. Together with robust contrastive language-audio pretraining (CLAP) representations, Make-An-Audio achieves state-of-the-art results in both objective and subjective benchmark evaluation. Moreover, we present its controllability and generalization for X-to-Audio with "No Modality Left Behind", for the first time unlocking the ability to generate high-definition, high-fidelity audios given a user-defined modality input. Audio samples are available at https://Text-to-Audio.github.io

1. Introduction

Make-An-Audio addresses text-to-audio’s data scarcity and long-waveform modeling challenges with prompt enhancement, spectrogram representations, CLAP, and diffusion. It achieves state-of-the-art evaluations and generalizes to multiple user-defined modalities.

  • Text-to-audio generation is limited by scarce high-quality text-audio pairs and the complexity of modeling long continuous waveforms.Source: §1 · p. 1
  • Subjective and objective evaluations achieve new state-of-the-art text-to-audio results with natural, controllable synthesis and zero-shot generalization to Clotho.Source: §1 · p. 2
  • The model generalizes to text, audio, image, and video inputs for high-definition, high-fidelity audio generation with personalized transfer and fine-grained control.Source: Figure 1 · p. 1, §1 · p. 2
  • Make-An-Audio uses latent diffusion with a spectrogram autoencoder to model long continuous waveforms through compressed audio representations.Source: §1 · pp. 1–2
  • Pseudo prompt enhancement uses language-free audios to create orders-of-magnitude concept compositions and alleviate data scarcity.Source: §1 · pp. 1–2
  • CLAP representations provide textual understanding with computational efficiency, complementing the prompt-enhanced diffusion model.Source: §1 · pp. 1–2

2. Related Works

Related work establishes that audio generation trails visual generation because of data scarcity and long continuous signals. Make-An-Audio combines prompt enhancement, continuous spectrogram representations, and diffusion to address these constraints.

  • Text-to-audio progress lags behind visual generation because high-quality text-audio datasets are scarce and long continuous waveforms are difficult to model.Source: §2.2 · p. 2, §2.1 · p. 2
  • Prior systems include DiffSound’s discrete diffusion over VQ-VAE audio codes and AudioLM’s autoregressive generation from masked-language-model audio activations.Source: §2.2 · p. 2
  • AudioGen generates audio autoregressively, whereas Make-An-Audio uses pseudo prompt enhancement, CLAP, diffusion, and continuous spectrogram representations.Source: §2.2 · p. 2
  • Make-An-Audio’s overview includes pseudo prompt enhancement, a spectrogram autoencoder, CLAP-conditioned latent diffusion, and a neural vocoder.Source: Figure 2 · p. 3
  • Pseudo prompt enhancement combines expert distillation with dynamic reprogramming to construct varied audio-aligned concept compositions.Source: Figure 3 · p. 3
  • Spectrogram autoencoders and self-supervised representations support audio processing by transferring techniques associated with image-like spectrograms.Source: §2.3 · p. 3

3. Make-An-Audio

Make-An-Audio addresses text-to-audio generation’s data scarcity and long-waveform modeling challenges with pseudo prompt enhancement, spectrogram autoencoding, and latent diffusion. The framework uses CLAP representations and classifier-free guidance to support efficient, semantically aligned synthesis.

  • Make-An-Audio targets data scarcity and the difficulty of modeling long continuous waveforms in text-to-audio generation.Source: §3.1 · p. 3

Pseudo Prompt Enhancement

  • Pseudo prompt enhancement combines expert distillation with dynamic reprogramming to create audio-aligned prompts and varied concept compositions from language-free audio.Source: §3.2 · p. 3, §3.2.1 · p. 3, §3.2.2 · p. 4

Textual Representation

  • CLAP and T5-Large achieve similar benchmark results, while CLAP is more computationally efficient because it does not require offline language-model embedding computation.Source: §3.3 · p. 4

Audio Representation

  • The framework predicts self-supervised spectrogram representations rather than waveforms, using a spectrogram autoencoder whose encoder produces latents and decoder reconstructs mel-spectrograms.Source: §3.1 · p. 3, §3.4 · p. 4

Generative Latent Diffusion

  • Latent diffusion maps textual representations to latent audio representations through conditional diffusion steps and trains with mean squared error in noise space.Source: §3.5 · p. 4

Classifier-Free Guidance

  • Classifier-free guidance combines conditional and unconditional diffusion scores during sampling through a guidance scale s ≥1.Source: §3.6 · pp. 4–5

4. X-To-Audio: No Modality Left Behind

Make-An-Audio extends conditional diffusion beyond text to audio, image, and video inputs. Its personalized generation and inpainting variants provide controllability while exposing a faithfulness–realism trade-off in audio editing.

X-To-Audio

  • X-to-Audio generalizes Make-An-Audio to text, audio inpainting, and visual inputs for user-defined modality-conditioned audio generation.Source: §4 · p. 5

Personalized Text-To-Audio Generation

  • Personalized text-to-audio generation uses stochastic differential editing to denoise a partially noised input audio under a user-provided prompt.Source: §4.1 · p. 5
  • As the noising duration T increases, generated samples become more realistic but less faithful to the text caption.Source: §4.1 · p. 5

Audio Inpainting

  • Audio inpainting fills masked regions with new content, and Make-An-Audio is explicitly fine-tuned to reduce edge artifacts from information loss during sampling.Source: §4.2 · p. 5

Visual-To-Audio Generation

  • Visual-to-audio generation uses CLIP-guided text-to-audio modeling to bridge image or video inputs and audio representations without requiring large paired visual-audio datasets.Source: §4.3 · p. 6

5. Training and Evaluation

The system is trained on a large mixture of audio resources with pseudo-generated captions and evaluated using objective and human measures of audio quality and text-audio faithfulness. Evaluation includes FID, KL divergence, CLAP score, and MOS-based ratings.

Training Data

  • Training combines multiple audio datasets into approximately 3k hours and 1M audio-text pairs, with pseudo prompt enhancement applied to unannotated audio.Source: §5.1 · p. 6

Preprocessing

  • Preprocessing converts audio to 16kHz, pads clips to 10 seconds, and extracts 80 × 624 mel-spectrograms using specified FFT and hop sizes.Source: §5.1 · p. 6

Model Training

  • A continuous autoencoder compresses perceptual audio space into a 4-channel latent representation, while a U-Net text-conditional diffusion model is trained for 2M optimization steps.Source: §5.2 · p. 6

Evaluation Metrics

  • Objective evaluation measures audio fidelity with melception-based FID and KL divergence, while CLAP score measures reference-free audio-text alignment.Source: §5.3 · p. 6

Human Evaluation

  • Human evaluation reports MOS-Q for audio quality and MOS-F for text-audio alignment faithfulness on a 20-100 Likert scale with 95% confidence intervals.Source: §5.3 · p. 6

6. Results

Make-An-Audio achieves strong text-to-audio results across objective and subjective evaluation, while supporting inpainting, personalized generation, and image/video-to-audio synthesis. Experiments also examine representation choices, guidance trade-offs, and prompt enhancement.

Quantitative Evaluation

  • Make-An-Audio achieves the highest AudioCaption perceptual quality, with FID of 4.61 and KL of 2.79.Source: §6.1 · p. 6
  • Make-An-Audio achieves the highest CLAP score, with a 0.037 gap to ground-truth audio.Source: §6.1 · p. 6

Human Evaluation

  • MOS-Q of 72.5 and MOS-F of 78.6 indicate raters preferred the synthesis for audio naturalness and faithfulness.Source: §6.1 · p. 7

Audio Inpainting

  • Larger training masks improve perceptual quality, while frame-based masking consistently outperforms irregular masking at similar region sizes.Source: §6.1 · p. 7

Image/Video-to-Audio

  • Make-An-Audio generalizes to images and videos, generating high-fidelity audio spectrograms aligned with their semantic meanings.Source: §6.1 · p. 7

Personalized Generation

  • For personalized generation, t0 ∈[0.2, 0.5] balances faithful guidance with realistic generation, whereas increasing t0 can destroy speed, timbre, and energy.Source: §6.2 · p. 7

Prompt Enhancement

  • Dynamic reprogramming improves evaluation and substantially supports zero-shot generalization to Clotho by constructing diverse object compositions.Source: §6.3.2 · p. 8

7. Conclusion

The paper concludes that Make-An-Audio combines prompt enhancement and diffusion modeling to produce realistic, faithful audio and extends generation to user-defined modalities. It positions the system as a basis for future audio synthesis studies.

  • Make-An-Audio combines prompt enhancement with distill-then-reprogram training and contrastive pretraining for realistic, faithful text-to-audio synthesis.Source: §7 · p. 8
  • The model generates high-definition, high-fidelity audio from user-defined modality inputs, supporting personalized transfer and fine-grained control.Source: §7 · p. 8
  • The authors present Make-An-Audio as a basis for future audio synthesis studies.Source: §7 · p. 8

Appendices

The appendix section is represented only by the paper title, so it provides no substantive appendix findings to summarize.

  • The paper is titled Make-An-Audio: Text-To-Audio Generation with Prompt-Enhanced Diffusion.Source: Appendices · p. 12

A. Detailed Experimental Setup

The experimental setup combines diverse audio sources into a large audio-text training collection and uses standardized preprocessing, alignment evaluation, and reported model hyperparameters.

Dataset Analysis

  • The study reports statistics for dataset combinations and text-audio alignment CLAP scores averaged across single-label datasets.Source: Table 4 · p. 12, Table 5 · p. 12

Dataset

Model Configuration

  • The paper lists Make-An-Audio model hyperparameters in a dedicated table.Source: Table 6 · p. 13

C. Evaluation

The evaluation uses human ratings for audio naturalness and text-audio alignment, alongside a diffusion formulation that progressively noises data and reverses the process to recover samples.

Subjective evaluation

  • Mean opinion score tests ask raters to judge subjective audio naturalness on a 20-100 Likert scale.Raters are explicitly instructed to focus on audio quality and naturalness.Source: C. Evaluation · p. 13
  • Text-audio alignment tests ask raters whether an audio and prompt align faithfully, using completely, mostly, or somewhat responses on a 20-100 Likert scale.Source: C. Evaluation · p. 13
  • The subjective tests are crowd-sourced through Amazon Mechanical Turk, with ratings collected independently for model samples and reference audio.Participant compensation totaled about $750 at $8 per hour.Source: C. Evaluation · p. 13

Diffusion formulation

  • The forward diffusion process is a fixed Markov chain that adds small Gaussian noise and gradually converts data x0 into whitened latents xT under schedule β1, · · · , βT.Training optimizes a randomly selected timestep term with stochastic gradient descent.Source: D. Detailed Formulation of DDPM · p. 13
  • The reverse process is a θ-parameterized Markov chain from xT to x0 that removes the Gaussian noise introduced during diffusion.Source: D. Detailed Formulation of DDPM · pp. 13–14

E. Implementation Details

The implementation combines spectrogram-based diffusion modeling, prompt and modality conditioning, and ablations of autoencoder depth, datasets, and global conditions. It also details template-based dynamic reprogramming and discusses computational and societal boundaries.

Audio autoencoder ablations

  • Deeper audio representations, using 32 or 128 dimensions, provide more compression but may burden the U-Net through information deterioration.The comparison appears in Table 7’s audio-quality ablation study.Source: §E.1 · p. 14, Table 7 · p. 14

Text-to-audio model

  • The base model encodes text as K tokens and uses cross-attention to map language representations to mel-spectrograms.It is subsequently fine-tuned with 20% empty text sequences to support both conditional and unconditional spectrogram generation.Source: §E.2 · pp. 14–15

X-to-audio conditioning

  • Visual-to-audio synthesis uses a CLIP-guided text-to-audio model, while FSD50K class labels simplify training by avoiding multimodal prediction.The authors report that global CLIP conditions have limited control over faithful synthesis with high text-audio similarity.Source: §E.3 · p. 15

Global-condition ablations

  • Replacing FSD50K with AudioCaps significantly decreases faithfulness because dynamic concept compositions confuse global-condition models and multimodal distributions hinder controllable synthesis.These comparisons are reported in Table 8’s global-condition ablation study.Source: §E.3 · p. 15, Table 8 · p. 15

Dynamic reprogramming

  • Dynamic reprogramming replaces X with sampled data language and & with sampled event class labels across multiple natural-language templates.The template vocabulary varies verbs, adjectives, nouns, and quantifiers to form alternative prompt constructions.Source: F. Dynamic Reprogramming Templates · pp. 15–16

Constraints and future direction

  • Diffusion synthesis requires multiple iterative refinements, while latent diffusion typically requires more computation and may degrade with less training data.The paper identifies lightweight, fast diffusion models as a future direction for accelerating sampling.Source: H. Limitations · p. 16

Related papers

Loading 2301.12661v1…