Source-linked AI summary
MusicLM: Generating Music From Text
Andrea Agostinelli, Timo I. Denk, Zalán Borsos, Jesse Engel, Mauro Verzetti, Antoine Caillon, Qingqing Huang, Aren Jansen, Adam Roberts, Marco Tagliasacchi, Matt Sharifi, Neil Zeghidour, Christian Frank
TL;DR
MusicLM tackles the challenge of generating rich, coherent music from text under scarce paired audio-text data. It uses hierarchical autoregressive audio modeling with MuLan text conditioning, and achieves high-quality, caption-faithful 24 kHz music while extending to melody conditioning and releasing MusicCaps. The paper also identifies detailed captions, temporal ordering, and negations as cases where ground truth was preferred.
Problem
Generating rich music from a single text caption is difficult because paired audio-text data are scarce and sequence-wide captions weakly describe temporally structured music.
Method
MusicLM combines AudioLM’s multi-stage autoregressive modeling with MuLan embeddings for text conditioning and extends conditioning to input melodies.
Results
MusicLM produces high-quality 24 kHz music consistent over several minutes, remains faithful to text, and outperforms baselines on MusicCaps.
Takeaways & Limitations
MusicLM supports captioned music generation, text-guided rendering of hummed or whistled melodies, and evaluation through the publicly released 5.5k-pair MusicCaps dataset.
Takeaways & Limitations
Ground truth was preferred for extremely detailed captions, temporal ordering, or negations, and future work includes higher-level song structure and improved text conditioning and vocal quality.
Abstract
from arXiv · showhide
We introduce MusicLM, a model generating high-fidelity music from text descriptions such as "a calming violin melody backed by a distorted guitar riff". MusicLM casts the process of conditional music generation as a hierarchical sequence-to-sequence modeling task, and it generates music at 24 kHz that remains consistent over several minutes. Our experiments show that MusicLM outperforms previous systems both in audio quality and adherence to the text description. Moreover, we demonstrate that MusicLM can be conditioned on both text and a melody in that it can transform whistled and hummed melodies according to the style described in a text caption. To support future research, we publicly release MusicCaps, a dataset composed of 5.5k music-text pairs, with rich text descriptions provided by human experts.
1. Introduction
MusicLM addresses the challenge of generating rich, coherent music from text despite scarce paired audio-text data and weak sequence-wide captions. It combines hierarchical audio modeling with text conditioning, supports melody conditioning, outperforms prior systems, and introduces MusicCaps for evaluation.
- Motivation: Turning a single text caption into a rich music sequence with long-term structure and many stems remains an open challenge.Earlier caption-based audio models were limited to simple acoustic scenes over seconds.
- Motivation: Paired audio-text data are scarce, while sequence-wide captions weakly represent music’s melody, rhythm, timbre, instruments, and temporal structure.The paper identifies both data scarcity and the difficulty of describing temporally structured audio.
- Approach: MusicLM extends AudioLM’s multi-stage autoregressive generation with text conditioning using MuLan’s shared music-text embedding space.Audio-derived MuLan embeddings condition training, while text-derived embeddings condition inference.
- Results: Experiments show MusicLM outperforms Mubert and Riffusion in audio quality and adherence to captions.The comparison uses quantitative metrics and human evaluations.
- Extensions and resources: The method supports text-plus-melody conditioning and the authors release MusicCaps, a hand-curated dataset of 5.5k music-text pairs prepared by musicians.Melody inputs can be whistled, hummed, sung, or played on an instrument; demonstrated clips reach up to five minutes.
2. Background and Related Work
MusicLM builds on discrete-token autoregressive audio generation and joint music-text embeddings, extending AudioLM toward text- and melody-conditioned music across diverse genres.
- Quantization: Autoregressive discrete-token modeling supports high-fidelity reconstruction across continuous signals such as audio, images, and video.Quantization provides compact discrete representations while retaining reconstruction quality.
- AudioLM: AudioLM separates semantic tokens for long-term structure from acoustic tokens for fine audio detail.This hierarchical representation addresses the trade-off between temporal coherence and synthesis quality.
- MusicLM: MusicLM adds descriptive text conditioning, supports melody conditioning, and models long music sequences beyond piano music.The reported genres include drum’n’bass, jazz, and classical music.
- Text-Conditioned Generation: Unlike DALL·E 2, MusicLM uses an AudioLM-based decoder and omits a prior mapping text embeddings to music embeddings.The decoder can therefore be trained on audio-only data, with text embeddings substituted during inference.
- Joint Embedding Models: MuLan maps music and text through two modality-specific towers into a shared 128-dimensional embedding space using contrastive learning.Its weakly associated training pairs support cross-modal correspondence for applications such as retrieval and zero-shot tagging.
3. Method
MusicLM combines pretrained audio representations with MuLan conditioning in a hierarchical sequence-to-sequence architecture, using separate autoregressive stages for semantic and acoustic modeling.
- Representation and Tokenization: MusicLM uses SoundStream acoustic tokens, w2v-BERT semantic tokens, and MuLan embeddings for conditioning.MuLan audio embeddings condition training, while MuLan text embeddings condition inference.
- Representation and Tokenization: SoundStream represents 24 kHz monophonic audio with 12 residual quantizers and 600 acoustic tokens per second.The representation uses a 6 kbps bitrate and 1024-token vocabularies for each quantizer.
- Conditioning: MuLan audio conditioning enables audio-only training and improves robustness to noisy text descriptions.The shared embedding space removes the need for captions during training while preserving text conditioning at inference.
- Hierarchical Modeling: The acoustic stage predicts acoustic tokens conditioned on both MuLan audio tokens and semantic tokens.It models p(A_t|A_<t, S, M_A) after semantic modeling.
- Hierarchical Modeling: To avoid long token sequences, acoustic modeling is split into coarse and fine stages.The coarse stage models four SoundStream RVQ levels, while the fine stage models the remaining eight.
4. Experimental Setup
The experimental setup trains MusicLM components on large audio-only corpora, evaluates quality and text adherence with complementary metrics and human ratings, and studies memorization.
- Models and Training: MusicLM’s semantic and acoustic autoregressive stages use decoder-only Transformers with 430M parameters per stage.Each model has 24 layers, 16 attention heads, 1024-dimensional embeddings, and 4096-dimensional feed-forward layers.
- Models and Training: The autoregressive models are trained on five million clips totaling 280k hours of music at 24 kHz.SoundStream and w2v-BERT are trained on the Free Music Archive, while other components use the larger audio-only corpus.
- Dataset: MusicCaps contains 5.5k AudioSet music clips paired with English descriptions written by ten professional musicians.Each 10-second clip includes an average four-sentence caption and music-aspect annotations.
- Dataset: MusicCaps focuses exclusively on music and provides detailed expert annotations alongside a genre-balanced split.It complements AudioCaps, which includes non-music content.
- Metrics: Evaluation metrics target two aspects of generation: audio quality and adherence to the text description.The setup includes FAD, KLD, MCC, and qualitative pairwise human ratings.
- Metrics: FAD is computed with Trill2 and VGGish to measure different aspects of audio quality.Trill2 is trained on speech data, whereas VGGish is trained on YouTube-8M audio events.
- Memorization Study: The memorization study adapts Carlini et al.’s methodology and focuses on MusicLM’s semantic modeling stage.It examines whether generated semantic-token sequences resemble corresponding training examples.
5. Results
MusicLM is evaluated against Mubert and Riffusion on MusicCaps using audio-quality, text-faithfulness, and human-listening measures. Results also examine memorization, semantic-token ablations, and diversity across acoustic and semantic sampling.
- Comparison to baselines: MusicLM is compared with Mubert and Riffusion on MusicCaps using FAD, KLD, MCC, and pairwise human listening tests.The evaluation covers both audio quality and faithfulness to descriptive captions.
- Comparison to baselines: MusicLM achieves better FADVGG scores than Mubert and Riffusion, while its FADTrill score is 0.44 versus Mubert’s 0.45 and Riffusion’s 0.76.The authors characterize MusicLM’s generated music as high-quality and comparable to Mubert under these metrics.
- Comparison to baselines: Human raters clearly prefer MusicLM over both baselines, although the ground-truth reference remains preferred in some comparisons.The listening test collected 1200 ratings across pairwise comparisons.
- Error patterns: Captions with more than five instruments, temporal ordering, or negations are patterns in which ground-truth music is preferred over MusicLM.The authors specifically associate these cases with detailed captions, ordered events, and negations that MuLan does not capture well.
- Ablations: Removing semantic modeling worsens text-faithfulness scores: KLD increases from 1.01 to 1.05 and MCC decreases from 0.51 to 0.49.FAD remains comparable in the ablation, while long-term structure also degrades.
- Token information: Fixing semantic and MuLan tokens yields diverse acoustic properties, whereas sampling semantic tokens as well produces greater diversity in melodies and rhythmic properties while remaining text-coherent.The observed shared properties include genre, rhythmical characteristics, and parts of the main melody.
- Memorization analysis: Exact semantic-token matches remain below 0.2% even with a 10-second prompt, while approximate matches concentrate in low-diversity sequences.Approximate-match sequences below a 0.5 matching score have average empirical entropy of 1.0 bits versus 4.6 bits for sampled semantic tokens.
6. Extensions
MusicLM is extended to condition generation on both text and an input melody, and to produce longer sequences with changing textual context. These extensions preserve melody adherence and support smooth, tempo-consistent semantic transitions.
- Melody conditioning: Melody conditioning uses paired audio with matching melodies but different acoustics to train a joint embedding model.Pairs include covers, instrumentals, vocals, and human humming or singing.
- Melody conditioning: MusicLM quantizes melody embeddings with RVQ and concatenates melody tokens with MuLan tokens for conditioning.At inference, melody tokens from input audio are combined with MuLan text tokens.
- Melody conditioning: The extended model generates music that follows humming, singing, whistling, or instrumental melodies while adhering to the text description.The melody and text jointly specify the generated music.
- Long generation: Autoregressive temporal generation enables sequences longer than the 30-second training sequences by reusing a 15-second prefix to generate another 15 seconds.The same text description conditions each extension.
- Story mode: Story mode changes the text conditioning every 15 seconds and produces smooth, tempo-consistent, semantically plausible transitions.The music context changes according to the successive text descriptions.
7. Conclusions
MusicLM generates high-quality, text-faithful music at 24 kHz with consistency over several minutes and outperforms baselines on the 5.5k-example MusicCaps dataset. The authors identify limitations inherited from MuLan and outline future work on conditioning, vocals, song structure, and sample rate.
- Conclusion: MusicLM produces high-quality music at 24 kHz that remains consistent over several minutes while following text conditioning.The conclusion presents these properties as central characteristics of the model.
- Conclusion: MusicLM outperforms baselines on MusicCaps, a hand-curated dataset of 5.5k music-text pairs prepared by musicians.The dataset is described as high-quality and released with the paper.
- Limitations: MuLan-related limitations include misunderstanding negations and failing to adhere to precise temporal ordering in text descriptions.The authors also state that MCC evaluations are favorable to MusicLM because MCC relies on MuLan.
- Future work: Future work includes lyrics generation, improved text conditioning and vocal quality, high-level song structure modeling, and higher sample-rate generation.The proposed song structures include introduction, verse, and chorus.
8. Broader Impact
MusicLM extends creative music tools while raising concerns about training-data bias, cultural appropriation, and creative-content misappropriation.
- Generated samples reflect biases in the training data, raising concerns about appropriateness for underrepresented cultures and cultural appropriation.
- The authors study memorization because music generation may misappropriate creative content, adapting a methodology for text-based language models.
- Only a tiny fraction of examples was memorized exactly, while approximate matches were identified for 1% of examples.
- The authors state that they have no plans to release models at this point.
A. MusicCaps Dataset
MusicCaps is a publicly released, expert-curated dataset for evaluating text-to-music generation, with 5,521 examples and a genre-balanced 1,000-example subset.
- 5,521 MusicCaps examples pair AudioSet music clips with English text descriptions; 2,858 come from AudioSet eval and 2,663 from train.
- A genre-balanced subset contains 1,000 examples, all drawn from the AudioSet evaluation split.
- Figure 4 shows genre distribution for all 5.5k examples, while Figure 5 shows the distribution for the balanced 1k subset.
B. Qualitative Evaluation
The qualitative evaluation used controlled pairwise listener comparisons of generated clips and references against text captions, with statistically significant model orderings.
- Listeners compared two 10-second clips against a text caption on a 5-point Likert scale, judging caption match while ignoring audio quality.
- The study collected 1,200 ratings, with each source involved in 600 pairwise comparisons.
- Wilcoxon signed-rank tests with Bonferroni correction found all orderings shown in Figure 8 statistically significant at p < 0.01/15.
- Figure 7 presents pairwise preference results; raters showed decisive preferences in every comparison except Mubert versus Riffusion.
- Figure 8 reports win percentages for caption matching across systems and against each system individually, using N = 1200 and N = 600 comparisons.
C. Melody Conditioning
The melody-conditioning model encodes temporal mel-spectrogram frames with a compact vision transformer trained using semi-hard triplet loss.
- The melody-conditioning model uses a 12-layer ViT with 6 attention heads, 512-dimensional embeddings, and 1024-dimensional feed-forward layers.
- Its input is the temporal frames of an audio mel spectrogram, and semi-hard triplet loss trains 192-dimensional melody embeddings.