Source-linked AI summary
Noise2Music: Text-conditioned Music Generation with Diffusion Models
Qingqing Huang, Daniel S. Park, Tao Wang, Timo I. Denk, Andy Ly, Nanxin Chen, Zhengdong Zhang, Zhishuai Zhang, Jiahui Yu, Christian Frank, Jesse Engel, Quoc V. Le, William Chan, Zhifeng Chen, Wei Han
TL;DR
Noise2Music addresses text-conditioned music generation by training cascaded diffusion models on large-scale music-text data. It generates intermediate representations and refines them into high-fidelity 30-second clips, with generated audio reflecting both standard musical attributes and fine-grained prompt semantics. The work also introduces MuLaMCap and evaluates audio quality and text-audio alignment.
Problem
High-quality paired music-text data describing musical content beyond metadata such as titles, artists, albums, and release years is scarce.
Method
Noise2Music trains cascaded diffusion models whose text-conditioned generators create spectrogram or low-fidelity-audio representations for subsequent high-fidelity audio generation, using pseudo-labeled data.
Results
The models generate 30-second, 24kHz clips that reflect musical attributes and fine-grained semantics such as atmosphere, feeling, and activity.
Takeaways & Limitations
MuLaMCap provides O(400K) music-text pairs with a diverse vocabulary for applications including music captioning, retrieval, and generation.
Takeaways & Limitations
The authors identify improving interpretability, text-audio alignment, generation length, and training and inference cost as future work.
Abstract
from arXiv · showhide
We introduce Noise2Music, where a series of diffusion models is trained to generate high-quality 30-second music clips from text prompts. Two types of diffusion models, a generator model, which generates an intermediate representation conditioned on text, and a cascader model, which generates high-fidelity audio conditioned on the intermediate representation and possibly the text, are trained and utilized in succession to generate high-fidelity music. We explore two options for the intermediate representation, one using a spectrogram and the other using audio with lower fidelity. We find that the generated audio is not only able to faithfully reflect key elements of the text prompt such as genre, tempo, instruments, mood, and era, but goes beyond to ground fine-grained semantics of the prompt. Pretrained large language models play a key role in this story -- they are used to generate paired text for the audio of the training set and to extract embeddings of the text prompts ingested by the diffusion models. Generated examples: https://google-research.github.io/noise2music
1. Introduction
Noise2Music uses cascaded diffusion models and large-scale pseudo-labeled music-text data to generate 30-second, 24kHz music from text prompts. The models reflect both conventional musical attributes and finer-grained semantic descriptions.
- Noise2Music generates 30-second, 24kHz music clips from text prompts using a diffusion-based method.
- The system trains cascading diffusion models in which a text-conditioned generator produces an intermediate representation and a cascader produces higher-fidelity audio.The intermediate representation can be a log-mel spectrogram or a 3.2kHz waveform, while the cascader generates 16kHz audio.
- The training pipeline constructs diverse music-text pairs by combining a large language model with a pretrained music-text joint embedding model.The language model proposes caption candidates, which the joint embedding model scores against unlabeled music clips.
- MuLaMCap contains O(400K) music-text pairs with captions drawn from a vocabulary of 4 million descriptive sentences and phrases.The captions provide greater diversity and granularity than the 632-class AudioSet ontology, whose music-related subset contains 141 classes.
- Evaluation uses Fréchet Audio Distance for generated-audio quality and MuLan similarity for semantic alignment between prompts and generated clips.
- The models handle genre, instruments, and era while also reflecting fine-grained soft attributes such as atmosphere, feeling, and activity.
2. Related Work
The related work places Noise2Music at the intersection of large-scale generative modeling, diffusion, and text-conditioned audio generation. It distinguishes prior music systems by their generation mechanism and representation of conditional signals.
- Recent generative models have scaled datasets and model capabilities across text, speech, images, and audio.
- Diffusion models generate high-quality images, audio, and video, while cascaded diffusion progressively refines lower-fidelity outputs into higher-fidelity results.
- Text-to-audio research has studied spectrogram generation and spectrogram-conditioned audio, alongside autoregressive and discrete-audio-code approaches.
- For music generation, Jukebox, Mubert, and MusicLM use autoregressive approaches, whereas Riffusion uses diffusion for spectrogram generation.
- Conditional signals are either projected into predefined interpretable spaces such as artist or genre vocabularies, or communicated through other learned representations.
- Concurrent research with overlap appeared while this work was being completed.
3. Methods
Noise2Music trains diffusion models to denoise audio-related representations under text and other conditioning signals, using cascaded generators and high-fidelity audio models. Its training data are expanded through pseudo-labeling to capture fine-grained musical semantics.
- 3.1. Diffusion models: Diffusion training predicts the noise vector from a corrupted sample, conditioning signal, and randomly sampled diffusion time.The corrupted sample is produced by a Gaussian process governed by a noise schedule.
- 3.1. Diffusion models: At inference, random noise is iteratively denoised using ancestral sampling, with stochasticity and denoising schedules available as controls.
- 3.1. Diffusion models: Classifier-free guidance combines conditional and unconditional noise predictions to improve alignment with the conditioning input.The guidance scale w is greater than 1, and dynamic clipping is applied.
- 3.2. Architecture: The 1D Efficient U-Net uses down-sampling and up-sampling blocks with residual connections, attention layers, and multiple routes for conditioning.Text enters through cross-attention, while time and aligned compressed inputs interact through other U-Net pathways.
- 3.3. Music generation models: Cascaded models generate an intermediate representation from text and then produce final audio conditioned on that representation, using either low-fidelity audio or spectrograms.
- 3.3. Music generation models: A spectrogram-conditioned vocoder and a final lightweight cascader convert intermediate or 16kHz outputs into 16kHz and 24kHz audio, respectively.The vocoder uses only the spectrogram, while the final cascader does not use text conditioning.
- 3.5. Pseudo labeling for music audio: Pseudo-labeling combines MuLan and LaMDA to assign fine-grained semantic labels to otherwise unlabeled music clips.The method uses large caption vocabularies that differ from standard music-classification labels in scale and semantic granularity.
4. Experiments and Results
The experiments describe the multi-model training and inference setup, evaluate audio quality and text-audio alignment, and examine how inference choices affect generation. Human listening tests find the waveform model comparable to MusicLM, while ablations reveal metric and parameter trade-offs.
- Model training details: Four 1D U-Net models implement waveform and spectrogram generation and cascaded high-fidelity audio synthesis.The models comprise a waveform generator and cascader, plus a spectrogram generator and vocoder.
- Model training details: The generator models train on full 30-second representations, whereas cascaders and vocoders train on randomly sampled 3–4-second snippets.Generator models use self-attention; cascaders and the vocoder do not.
- Model inference: Inference cost is measured on four TPU V4 devices while producing four samples per request, with GSPMD reducing serving time by more than 50%.The models are served on Google Cloud TPU V4.
- Evaluation metrics: FAD measures generated-audio proximity to reference audio, while MuLan similarity measures semantic alignment between generated audio and text or corresponding audio.FAD uses embeddings from VGG, Trill, or MuLan encoders; MuLan similarity is computed as cosine similarity.
- Evaluation results: The evaluation metrics require caution because training-data overlap with evaluation sets and MuLan-based pseudo-labeling may advantage the reported models over baselines.The metrics remain quantitative measures of trained-model performance in the AudioSet domain.
- Evaluation results: Human listening tests collect 3k ratings and find the waveform model comparable to MusicLM, though behind ground-truth audio.Each source participates in 1.2k pairwise comparisons using MusicCaps captions and 10-second clips.
- Inference parameter ablations: Ablations show overall FAD–similarity correlation, but cascader FAD can worsen as similarity improves; excessive CFG harms quality and cascader scheduling has especially large impact.Generator CFG scale matters more than the generator denoising schedule.
5. Qualitative analysis
The models ground diverse prompt attributes in generated music, but often struggle with high-quality audio for out-of-distribution prompts.
- Generated music manifests the prompt’s genre, instrument, mood, vocal traits, and era.
- The models often struggle to produce high-quality audio from out-of-distribution prompts, despite generating some interesting examples.
6. Discussion
The discussion contrasts spectrogram and waveform intermediate representations and identifies several directions for improving the system beyond this work.
- Spectrogram vs. waveform approach: Spectrogram models are cheaper to train and serve, scale better with time length, and retain high-frequency information absent from low-fidelity audio.
- Future directions: Future improvements include interpretability, text-audio alignment, training and inference cost, and generation length.
7. Broader Impact
The paper discusses potential benefits and risks of text-conditioned music generation, including co-creation opportunities, dataset and labeling boundaries, bias, misuse, and withheld release pending clearer safety understanding.
- The work could become a useful tool for artists and content creators, but meaningful co-creation requires more work with musicians and other stakeholders.
- Large generative models can propagate biases from their text and music corpora, including subtle and unpredictable harms not fully captured by current benchmarks.
- Training data cover a limited and uneven corpus of global musical samples and genres, while labels can essentialize genres without community participation.
- Created content may be misappropriated when it exactly matches training examples, so duplication checks are built into the release pipeline.
- The authors do not intend to release the model until potential safety issues and risks are better understood.
A.1. Noise schedules
The paper uses linear and cosine noise schedules, with the cosine schedule implemented as a slight variant whose parameters set the endpoints of λ_t.
- Two noise schedules are used: the linear schedule from Ho et al. and the cosine schedule from Nichol and Dhariwal.
- The linear schedule makes σ2_t increase linearly with time.
- The cosine schedule is implemented as a slight variant of the schedule proposed by Nichol and Dhariwal.
- The cosine parameters a and b are selected so that λ_t starts at 20 and ends at −20.
A.2. Denoising step schedules
Inference uses hand-tuned denoising step schedules for the generator and cascader/vocoder models, with final evaluation schedules distinguished from ablation schedules.
- Denoising step schedules are specified separately for the generator and cascader/vocoder models.
- The schedules were hand-tuned by trial and error, and their numerical step sizes are listed in Table 8.
- Final evaluation schedules are italicized, and the spectrogram generator’s back-heavy schedule is further modified for evaluation.
B. Model architecture
The models use one-dimensional U-Nets organized as mirrored down-sampling and up-sampling paths with convolutional blocks between them.
- The U-Net applies an entry convolution, repeated down-sampling and up-sampling layers, and convolutional blocks between these paths.
- Table 8 lists denoising time-step values and italicizes the schedules selected for each model.
- Figure 4 describes the denoising-time trajectories for four schedules, distinguishing front-heavy schedules from back-heavy schedules.
B.1. Down-sampling and up-sampling blocks
The U-Net uses depth-indexed one-dimensional down-sampling and up-sampling layers with uniform convolutional blocks, configurable attention, and several architectural hyperparameters.
- Down-sampling and up-sampling: At depth I, each down-sampling layer uses a one-dimensional convolution with stride S_I to transform feature length and channel dimensions.
- Down-sampling and up-sampling: The same uniform convolutional block is used in both down-sampling and up-sampling paths, with matching block counts at each depth.
- Down-sampling and up-sampling: The entry and exit convolutions use kernel size 3, while the base model dimension also sets the zeroth channel width and time-embedding dimension.
- Attention and fixed components: Self- and cross-attention are optional by depth, and the number of attention heads is configurable; the remaining architecture is fixed.
- Hyperparameters: The architecture is governed by convolutional kernel size, network depth D, down-sampling factors, and the number of blocks at each depth.
B.2. Model specifications
The section identifies the architectural parameters used in the work and provides descriptive examples of music used to characterize genre, mood, instrumentation, and other musical attributes.
- Table 9 summarizes the architectural parameters of the models used in this work.
- The examples span songs described by genre, mood, tempo, instrumentation, vocal characteristics, and production techniques.
- Several examples characterize songs through combinations of rhythm sections, guitars, keyboards, synthesizers, strings, horns, and distinctive vocal performances.
- The listed metadata also includes licensing information for AudioSet labels and associated label identifiers.