Source-linked AI summary
Simple and Controllable Music Generation
Jade Copet, Felix Kreuk, Itai Gat, Tal Remez, David Kant, Gabriel Synnaeve, Yossi Adi, Alexandre Défossez
TL;DR
Conditional music generation requires controllable modeling of complex musical structure from text or melody. MusicGen uses a single-stage transformer over interleaved compressed audio-token streams, and the authors report high-quality mono and stereo generation that outperforms evaluated baselines. The paper also identifies efficiency, conditioning, and scope trade-offs through empirical studies and discusses limitations in fine-grained adherence control and audio-conditioning augmentation.
Problem
Text-conditioned music generation must model long, full-spectrum, harmonically complex sequences while giving creators control over attributes such as melody, instruments, and genre.
Method
MusicGen uses one autoregressive transformer over EnCodec’s parallel discrete audio-token streams, with codebook interleaving and text or unsupervised chromagram-based melody conditioning.
Results
84.8 out of 100 versus 80.5 for the best baseline is MusicGen’s reported subjective rating, while human evaluation also finds better melodic alignment and text adherence.
Takeaways & Limitations
A single-stage interleaved-token model can provide high-quality, text- and melody-conditioned music generation, including stereo output without extra computational cost.
Takeaways & Limitations
The method does not provide fine-grained control over conditioning adherence and requires further research on data augmentation, guidance, and audio conditioning.
Abstract
from arXiv · showhide
We tackle the task of conditional music generation. We introduce MusicGen, a single Language Model (LM) that operates over several streams of compressed discrete music representation, i.e., tokens. Unlike prior work, MusicGen is comprised of a single-stage transformer LM together with efficient token interleaving patterns, which eliminates the need for cascading several models, e.g., hierarchically or upsampling. Following this approach, we demonstrate how MusicGen can generate high-quality samples, both mono and stereo, while being conditioned on textual description or melodic features, allowing better controls over the generated output. We conduct extensive empirical evaluation, considering both automatic and human studies, showing the proposed approach is superior to the evaluated baselines on a standard text-to-music benchmark. Through ablation studies, we shed light over the importance of each of the components comprising MusicGen. Music samples, code, and models are available at https://github.com/facebookresearch/audiocraft
1 Introduction
Conditional music generation must model long, complex musical sequences while preserving harmony, melody, and controllability. MusicGen addresses this with a single-stage model and efficient codebook interleaving, supporting text and melody conditioning while outperforming evaluated baselines in human evaluation.
- Motivation: Text-to-music generates musical pieces from descriptions, but music demands long-range modeling, full-spectrum audio, harmonic structure, and precise melodic control.The paper highlights higher music sampling rates than speech, multiple interacting instruments, listener sensitivity to disharmony, and creator needs for controls such as key, instruments, melody, and genre.
- Contribution: MusicGen models multiple parallel streams of acoustic tokens with a general codebook interleaving framework instead of relying on cascaded models.The framework extends generation to stereo audio without extra computational cost.
- Contribution: MusicGen supports both textual and unsupervised melody conditioning, generating audio aligned with harmonic and melodic structure while following text descriptions.The model uses a single model for text- and melody-conditioned generation.
- Results: 84.8 out of 100 versus 80.5 for the best baseline is the reported subjective rating for MusicGen in the paper’s evaluation.The authors report extensive objective and human studies and describe MusicGen as superior to evaluated baselines by a large margin.
2 Method
MusicGen combines EnCodec’s parallel discrete audio tokens with a transformer decoder that predicts them through configurable codebook interleaving patterns. Text or melody representations condition the same model, enabling efficient generation while exposing trade-offs between exactness, speed, and controllability.
- Model: MusicGen uses an autoregressive transformer decoder over EnCodec tokens, conditioned on either text or melody representations.The decoder uses codebook-specific embeddings and logits within the chosen pattern.
- Audio tokenization: EnCodec converts audio into K parallel discrete token sequences using residual vector quantization, whose codebooks represent successive quantization errors.The first codebook is the most important, and the streams are generally dependent.
- Codebook decomposition: Flattening all codebooks can theoretically model the token distribution exactly but increases the number of autoregressive steps to d · f_r · K.The increased complexity reduces part of the efficiency gained from the lower audio frame rate.
- Codebook decomposition: Inexact decompositions predict some codebooks in parallel, retaining the original frame rate and potentially speeding training and inference for long sequences.The approximation can diverge from the true distribution because parallel codebooks are not generally conditionally independent, causing errors to compound as t increases.
- Codebook interleaving patterns: Codebook interleaving patterns partition time-step and codebook positions into sequential groups whose positions are predicted in parallel.The paper evaluates parallel, delayed, and other patterns to measure the effects of inexact decomposition.
- Model conditioning: Text conditioning uses a representation tensor for the transformer, while melody conditioning combines text with a bottlenecked chromagram derived from dominant time-frequency bins.The bottleneck reduces overfitting observed with raw chromagrams, and the melody approach does not require supervised proprietary data.
3 Experimental setup
The experimental setup trains MUSICGEN on licensed music and evaluates it with objective and human metrics against established text-to-music baselines. Experiments cover text and melody conditioning on MusicCaps and held-out music data.
- Models and hyperparameters: The models use EnCodec tokenization, autoregressive transformers of 300M–3.3B parameters, T5 text conditioning, and the delay codebook pattern.Final text preprocessing uses condition-merging, text-description dropout, and word dropout; melody conditioning is optional.
- Datasets: MUSICGEN is trained on 20K hours of licensed music, including internal, ShutterStock, and Pond5 collections.The tracks contain textual descriptions and metadata such as genre, BPM, and tags; audio is downmixed to mono unless otherwise stated.
- Datasets: MusicCaps provides 5.5K ten-second samples and a 1K genre-balanced subset for main comparisons and qualitative evaluation.Objective metrics use the unbalanced set, while qualitative examples are sampled from the balanced subset; melody and ablations use 528 held-out in-domain tracks.
- Baselines: The study compares MUSICGEN with Riffusion and Mousai, and additionally uses reported or public-demo results for MusicLM and Noise2Music.Mousai is retrained on the study’s dataset, while MusicLM human tests use its public API and Noise2Music contributes reported FAD on MusicCaps.
- Evaluation: Evaluation combines FAD, KL divergence, and CLAP with human ratings of overall quality and text relevance on 1–100 scales.FAD is interpreted such that lower scores indicate more plausible generated audio.
4 Results
MUSICGEN outperforms evaluated baselines in human judgments of audio quality and text adherence, while objective metrics reveal trade-offs for melody conditioning. Ablations favor efficient delay-based token interleaving and show scale-dependent gains.
- 4.1 Comparison with the baselines: MUSICGEN surpasses evaluated baselines in human ratings of audio quality and adherence to text descriptions.Noise2Music achieves the best reported FAD on MusicCaps, followed by text-conditioned MUSICGEN.
- 4.2 Melody evaluation: Melody conditioning worsens objective metrics but does not significantly change human ratings and remains superior to evaluated baselines.The melody-conditioned system is evaluated with objective and subjective measures on held-out data.
- 4.3 Stereo generation: Stereo generation uses eight codebooks from independently encoded left and right channels and tests stereo delay variants from a pretrained monophonic model.The stereo model is fine-tuned on stereo audio, and one model is also evaluated after downmixing to mono.
- 4.4 Ablation: Flattening achieves the best codebook-pattern scores, but delay reaches similar performance at a fraction of flattening’s computational cost.Parallel performs worse, while delay and partial flattening achieve similar scores to one another.
- 4.4 Ablation: Scaling from 300M to 3.3B parameters improves objective scores but increases training and inference time; subjective quality peaks at 1.5B.The 3.3B model fits text prompts better despite no further improvement in subjective overall quality.
5 Related work
Related work represents audio or music with compressed discrete streams and applies generative models over those representations. Prior approaches include recurrent, GAN, hierarchical transformer, cascade, and diffusion-based systems across music and audio generation.
- Audio representation: Compressed discrete or continuous audio representations are commonly paired with generative models to make music and audio modeling tractable.Residual vector quantization and VQ-VAE approaches produce representations used by later generative models.
- Music generation: Earlier music-generation research spans GANs, unsupervised symbolic segmentation, recurrent neural networks, and surveys of deep-learning methods.These approaches primarily address symbolic or polyphonic music generation.
- Music generation: Hierarchical VQ-VAE systems and cascaded transformers generate music from multiple streams of semantic and acoustic representations.Related work also applies sparse transformers or cascaded transformer decoders over multistream representations.
- Music generation: Diffusion-based approaches generate text-conditioned music through latent diffusion, audio encoder-decoder diffusion, or cascades that increase sampling rate.These systems include methods for text-to-music and singing-to-accompaniment generation.
6 Discussion
MUSICGEN combines single-stage generation with text and melody conditioning, while the discussion identifies limits in fine-grained control, audio-conditioning augmentation, dataset diversity, and broader societal impact.
- MUSICGEN generates high-quality stereo audio with a single-stage model and fewer autoregressive steps than flattening.
- Text and melody conditioning provide controls for generating music that follows descriptions and melodic structure.
- Fine-grained adherence control remains limited because the method relies mostly on classifier-free guidance.
- Audio conditioning requires further research on data augmentation, guidance types, and guidance amounts.
- The dataset contains a larger proportion of western-style music, raising a potential lack-of-diversity concern.
- The discussion also identifies ethical challenges involving artists, legal rights, and access to generative models.
A Appendix
Figure A.1 contrasts partial flattening and partial delay for interleaving four parallel codebook streams across time.
- Partial flattening separates the first codebook into dedicated steps and interleaves it with parallel sampling of the other three codebooks.
- Partial flattening produces M interleaved steps, twice the N original time steps.
- Partial delay shifts codebooks 2, 3, and 4 by the same amount, using a delay of 1 in the example.
A.1 Experimental details
The appendix describes codebook interleaving, chromagram-based melody conditioning, and the dataset’s uneven genre distribution as key experimental details.
- Codebook interleaving patterns: Interleaving patterns exploit the greater importance of the first RVQ codebook while predicting remaining codebooks in parallel.
- Melody conditioning: Chromagram conditioning preserves melodic structure while enabling novel generated samples conditioned jointly on melody and text.
- Distribution of genres: The dataset is dominated by Dance/EDM, and oversampling underrepresented genres often worsened overall model performance.
A.2 Additional experimental results
Additional experiments examine encoders, text augmentation, audio tokenizers, optimization, genre distributions, and melody-conditioned generation, revealing several model- and setup-dependent trade-offs.
- Text encoder: T5 and Flan-T5 perform similarly on objective metrics, while the CLAP-based encoder performs worse except on CLAP score.
- Text augmentations: Condition merging improves FAD and KL, whereas text normalization and word dropout do not improve objective metrics.
- Audio tokenizer: The DAC comparison uses a vocal-free dataset, so its objective metrics do not match those in other tables.
- Audio tokenizer: DAC operates at 86 Hz while MUSICGEN is designed for 50 Hz, reducing inference runtime by 40% for the same audio.
- Optimization: D-Adaptation improves convergence for the 300M model but deteriorates train and validation performance for larger 1.5B and 3.3B models.
- Melody conditioning: Chromagram-conditioned samples strongly adhere to input melodies while text conditioning guides their generated style.
- Genre distribution: The training dataset’s top genres are unevenly distributed, with Dance/EDM prominent.