Source-linked AI summary
Make-An-Audio: Text-To-Audio Generation with Prompt-Enhanced Diffusion Models
Rongjie Huang, Jiawei Huang, Dongchao Yang, Yi Ren, Luping Liu, Mingze Li, Zhenhui Ye, Jinglin Liu, Xiang Yin, Zhou Zhao
TL;DR
Text-to-audio generation is constrained by scarce high-quality text-audio pairs and the difficulty of modeling long continuous audio. Make-An-Audio addresses these gaps with pseudo prompt enhancement, spectrogram autoencoding, CLAP, and diffusion, achieving state-of-the-art evaluations and multimodal X-to-Audio generalization.
Problem
Text-to-audio generation lacks large-scale high-quality text-audio datasets and requires modeling complex long continuous waveforms.
Method
Make-An-Audio combines distill-then-reprogram pseudo prompt enhancement, spectrogram autoencoding, CLAP representations, and latent diffusion for text-to-audio generation.
Results
Make-An-Audio achieves new state-of-the-art text-to-audio results in objective and subjective evaluations, with realistic faithful synthesis and multimodal generalization.
Takeaways & Limitations
The model generates high-definition, high-fidelity audio from user-defined text, audio, image, and video inputs.
Takeaways & Limitations
Diffusion requires multiple iterative refinements, while latent diffusion typically requires more computational resources and can degrade with less training data.
Abstract
from arXiv · showhide
Large-scale multimodal generative modeling has created milestones in text-to-image and text-to-video generation. Its application to audio still lags behind for two main reasons: the lack of large-scale datasets with high-quality text-audio pairs, and the complexity of modeling long continuous audio data. In this work, we propose Make-An-Audio with a prompt-enhanced diffusion model that addresses these gaps by 1) introducing pseudo prompt enhancement with a distill-then-reprogram approach, it alleviates data scarcity with orders of magnitude concept compositions by using language-free audios; 2) leveraging spectrogram autoencoder to predict the self-supervised audio representation instead of waveforms. Together with robust contrastive language-audio pretraining (CLAP) representations, Make-An-Audio achieves state-of-the-art results in both objective and subjective benchmark evaluation. Moreover, we present its controllability and generalization for X-to-Audio with "No Modality Left Behind", for the first time unlocking the ability to generate high-definition, high-fidelity audios given a user-defined modality input. Audio samples are available at https://Text-to-Audio.github.io
1. Introduction
Make-An-Audio addresses text-to-audio’s data scarcity and long-waveform modeling challenges with prompt enhancement, spectrogram representations, CLAP, and diffusion. It achieves state-of-the-art evaluations and generalizes to multiple user-defined modalities.
- Text-to-audio generation is limited by scarce high-quality text-audio pairs and the complexity of modeling long continuous waveforms.
- Subjective and objective evaluations achieve new state-of-the-art text-to-audio results with natural, controllable synthesis and zero-shot generalization to Clotho.
- The model generalizes to text, audio, image, and video inputs for high-definition, high-fidelity audio generation with personalized transfer and fine-grained control.
- Make-An-Audio uses latent diffusion with a spectrogram autoencoder to model long continuous waveforms through compressed audio representations.
- Pseudo prompt enhancement uses language-free audios to create orders-of-magnitude concept compositions and alleviate data scarcity.
- CLAP representations provide textual understanding with computational efficiency, complementing the prompt-enhanced diffusion model.
2. Related Works
Related work establishes that audio generation trails visual generation because of data scarcity and long continuous signals. Make-An-Audio combines prompt enhancement, continuous spectrogram representations, and diffusion to address these constraints.
- Text-to-audio progress lags behind visual generation because high-quality text-audio datasets are scarce and long continuous waveforms are difficult to model.
- Prior systems include DiffSound’s discrete diffusion over VQ-VAE audio codes and AudioLM’s autoregressive generation from masked-language-model audio activations.
- AudioGen generates audio autoregressively, whereas Make-An-Audio uses pseudo prompt enhancement, CLAP, diffusion, and continuous spectrogram representations.
- Make-An-Audio’s overview includes pseudo prompt enhancement, a spectrogram autoencoder, CLAP-conditioned latent diffusion, and a neural vocoder.
- Pseudo prompt enhancement combines expert distillation with dynamic reprogramming to construct varied audio-aligned concept compositions.
- Spectrogram autoencoders and self-supervised representations support audio processing by transferring techniques associated with image-like spectrograms.
3. Make-An-Audio
Make-An-Audio addresses text-to-audio generation’s data scarcity and long-waveform modeling challenges with pseudo prompt enhancement, spectrogram autoencoding, and latent diffusion. The framework uses CLAP representations and classifier-free guidance to support efficient, semantically aligned synthesis.
- Make-An-Audio targets data scarcity and the difficulty of modeling long continuous waveforms in text-to-audio generation.
- Pseudo Prompt Enhancement: Pseudo prompt enhancement combines expert distillation with dynamic reprogramming to create audio-aligned prompts and varied concept compositions from language-free audio.
- Textual Representation: CLAP and T5-Large achieve similar benchmark results, while CLAP is more computationally efficient because it does not require offline language-model embedding computation.
- Audio Representation: The framework predicts self-supervised spectrogram representations rather than waveforms, using a spectrogram autoencoder whose encoder produces latents and decoder reconstructs mel-spectrograms.
- Generative Latent Diffusion: Latent diffusion maps textual representations to latent audio representations through conditional diffusion steps and trains with mean squared error in noise space.
- Classifier-Free Guidance: Classifier-free guidance combines conditional and unconditional diffusion scores during sampling through a guidance scale s ≥1.
4. X-To-Audio: No Modality Left Behind
Make-An-Audio extends conditional diffusion beyond text to audio, image, and video inputs. Its personalized generation and inpainting variants provide controllability while exposing a faithfulness–realism trade-off in audio editing.
- X-To-Audio: X-to-Audio generalizes Make-An-Audio to text, audio inpainting, and visual inputs for user-defined modality-conditioned audio generation.
- Personalized Text-To-Audio Generation: Personalized text-to-audio generation uses stochastic differential editing to denoise a partially noised input audio under a user-provided prompt.
- Personalized Text-To-Audio Generation: As the noising duration T increases, generated samples become more realistic but less faithful to the text caption.
- Audio Inpainting: Audio inpainting fills masked regions with new content, and Make-An-Audio is explicitly fine-tuned to reduce edge artifacts from information loss during sampling.
- Visual-To-Audio Generation: Visual-to-audio generation uses CLIP-guided text-to-audio modeling to bridge image or video inputs and audio representations without requiring large paired visual-audio datasets.
5. Training and Evaluation
The system is trained on a large mixture of audio resources with pseudo-generated captions and evaluated using objective and human measures of audio quality and text-audio faithfulness. Evaluation includes FID, KL divergence, CLAP score, and MOS-based ratings.
- Training Data: Training combines multiple audio datasets into approximately 3k hours and 1M audio-text pairs, with pseudo prompt enhancement applied to unannotated audio.
- Preprocessing: Preprocessing converts audio to 16kHz, pads clips to 10 seconds, and extracts 80 × 624 mel-spectrograms using specified FFT and hop sizes.
- Model Training: A continuous autoencoder compresses perceptual audio space into a 4-channel latent representation, while a U-Net text-conditional diffusion model is trained for 2M optimization steps.
- Evaluation Metrics: Objective evaluation measures audio fidelity with melception-based FID and KL divergence, while CLAP score measures reference-free audio-text alignment.
- Human Evaluation: Human evaluation reports MOS-Q for audio quality and MOS-F for text-audio alignment faithfulness on a 20-100 Likert scale with 95% confidence intervals.
6. Results
Make-An-Audio achieves strong text-to-audio results across objective and subjective evaluation, while supporting inpainting, personalized generation, and image/video-to-audio synthesis. Experiments also examine representation choices, guidance trade-offs, and prompt enhancement.
- Quantitative Evaluation: Make-An-Audio achieves the highest AudioCaption perceptual quality, with FID of 4.61 and KL of 2.79.
- Quantitative Evaluation: Make-An-Audio achieves the highest CLAP score, with a 0.037 gap to ground-truth audio.
- Human Evaluation: MOS-Q of 72.5 and MOS-F of 78.6 indicate raters preferred the synthesis for audio naturalness and faithfulness.
- Audio Inpainting: Larger training masks improve perceptual quality, while frame-based masking consistently outperforms irregular masking at similar region sizes.
- Image/Video-to-Audio: Make-An-Audio generalizes to images and videos, generating high-fidelity audio spectrograms aligned with their semantic meanings.
- Personalized Generation: For personalized generation, t0 ∈[0.2, 0.5] balances faithful guidance with realistic generation, whereas increasing t0 can destroy speed, timbre, and energy.
- Prompt Enhancement: Dynamic reprogramming improves evaluation and substantially supports zero-shot generalization to Clotho by constructing diverse object compositions.
7. Conclusion
The paper concludes that Make-An-Audio combines prompt enhancement and diffusion modeling to produce realistic, faithful audio and extends generation to user-defined modalities. It positions the system as a basis for future audio synthesis studies.
- Make-An-Audio combines prompt enhancement with distill-then-reprogram training and contrastive pretraining for realistic, faithful text-to-audio synthesis.
- The model generates high-definition, high-fidelity audio from user-defined modality inputs, supporting personalized transfer and fine-grained control.
- The authors present Make-An-Audio as a basis for future audio synthesis studies.
Appendices
The appendix section is represented only by the paper title, so it provides no substantive appendix findings to summarize.
- The paper is titled Make-An-Audio: Text-To-Audio Generation with Prompt-Enhanced Diffusion.
A. Detailed Experimental Setup
The experimental setup combines diverse audio sources into a large audio-text training collection and uses standardized preprocessing, alignment evaluation, and reported model hyperparameters.
- Dataset Analysis: The study reports statistics for dataset combinations and text-audio alignment CLAP scores averaged across single-label datasets.
- Dataset: The dataset combines several publicly available audio sources and contains 1M audio-text pairs totaling approximately 3k hours.
- Dataset: Pseudo prompt enhancement constructs aligned natural-language descriptions for audio lacking text annotations.
- Dataset: Speech and music samples are filtered from AudioSet to create a more balanced dataset.
- Model Configuration: The paper lists Make-An-Audio model hyperparameters in a dedicated table.
C. Evaluation
The evaluation uses human ratings for audio naturalness and text-audio alignment, alongside a diffusion formulation that progressively noises data and reverses the process to recover samples.
- Subjective evaluation: Mean opinion score tests ask raters to judge subjective audio naturalness on a 20-100 Likert scale.Raters are explicitly instructed to focus on audio quality and naturalness.
- Subjective evaluation: Text-audio alignment tests ask raters whether an audio and prompt align faithfully, using completely, mostly, or somewhat responses on a 20-100 Likert scale.
- Subjective evaluation: The subjective tests are crowd-sourced through Amazon Mechanical Turk, with ratings collected independently for model samples and reference audio.Participant compensation totaled about $750 at $8 per hour.
- Diffusion formulation: The forward diffusion process is a fixed Markov chain that adds small Gaussian noise and gradually converts data x0 into whitened latents xT under schedule β1, · · · , βT.Training optimizes a randomly selected timestep term with stochastic gradient descent.
- Diffusion formulation: The reverse process is a θ-parameterized Markov chain from xT to x0 that removes the Gaussian noise introduced during diffusion.
E. Implementation Details
The implementation combines spectrogram-based diffusion modeling, prompt and modality conditioning, and ablations of autoencoder depth, datasets, and global conditions. It also details template-based dynamic reprogramming and discusses computational and societal boundaries.
- Audio autoencoder ablations: Deeper audio representations, using 32 or 128 dimensions, provide more compression but may burden the U-Net through information deterioration.The comparison appears in Table 7’s audio-quality ablation study.
- Text-to-audio model: The base model encodes text as K tokens and uses cross-attention to map language representations to mel-spectrograms.It is subsequently fine-tuned with 20% empty text sequences to support both conditional and unconditional spectrogram generation.
- X-to-audio conditioning: Visual-to-audio synthesis uses a CLIP-guided text-to-audio model, while FSD50K class labels simplify training by avoiding multimodal prediction.The authors report that global CLIP conditions have limited control over faithful synthesis with high text-audio similarity.
- Global-condition ablations: Replacing FSD50K with AudioCaps significantly decreases faithfulness because dynamic concept compositions confuse global-condition models and multimodal distributions hinder controllable synthesis.These comparisons are reported in Table 8’s global-condition ablation study.
- Dynamic reprogramming: Dynamic reprogramming replaces X with sampled data language and & with sampled event class labels across multiple natural-language templates.The template vocabulary varies verbs, adjectives, nouns, and quantifiers to form alternative prompt constructions.
- Constraints and future direction: Diffusion synthesis requires multiple iterative refinements, while latent diffusion typically requires more computation and may degrade with less training data.The paper identifies lightweight, fast diffusion models as a future direction for accelerating sampling.