Source-linked AI summary
Mustango: Toward Controllable Text-to-Music Generation
Jan Melechovsky, Zixun Guo, Deepanway Ghosal, Navonil Majumder, Dorien Herremans, Soujanya Poria
TL;DR
Text-to-music models have achieved strong generation quality, but control over musical attributes and access to paired music-text data remain limited. Mustango addresses these gaps with MuNet-guided diffusion and the augmented MusicBench dataset, and the paper reports state-of-the-art quality and stronger controllability through music-specific prompts. Its scope is limited to Western-music control, paired captions, and clips up to ten seconds.
Problem
Existing text-to-music systems provide limited control over musical attributes such as tempo, key, chords, and beats, while paired music-text datasets remain scarce.
Method
Mustango combines MuNet-guided latent diffusion with MusicBench, whose audio is diversified and whose captions are enriched with extracted beats, chords, key, and tempo.
Results
Mustango achieves state-of-the-art music quality and substantially improved controllability through music-specific prompts compared with systems including MusicGen and AudioLDM2.
Takeaways & Limitations
MusicBench and MuNet support text-to-music generation that follows musical properties embedded in user-specified prompts.
Takeaways & Limitations
Controllability is limited to Western music, requires paired text captions, and currently generates music up to ten seconds long.
Abstract
from arXiv · showhide
The quality of the text-to-music models has reached new heights due to recent advancements in diffusion models. The controllability of various musical aspects, however, has barely been explored. In this paper, we propose Mustango: a music-domain-knowledge-inspired text-to-music system based on diffusion. Mustango aims to control the generated music, not only with general text captions, but with more rich captions that can include specific instructions related to chords, beats, tempo, and key. At the core of Mustango is MuNet, a Music-Domain-Knowledge-Informed UNet guidance module that steers the generated music to include the music-specific conditions, which we predict from the text prompt, as well as the general text embedding, during the reverse diffusion process. To overcome the limited availability of open datasets of music with text captions, we propose a novel data augmentation method that includes altering the harmonic, rhythmic, and dynamic aspects of music audio and using state-of-the-art Music Information Retrieval methods to extract the music features which will then be appended to the existing descriptions in text format. We release the resulting MusicBench dataset which contains over 52K instances and includes music-theory-based descriptions in the caption text. Through extensive experiments, we show that the quality of the music generated by Mustango is state-of-the-art, and the controllability through music-specific text prompts greatly outperforms other models such as MusicGen and AudioLDM2.
1 Introduction
Mustango addresses limited control over musical structure in text-to-music generation and the scarcity of paired music-text data. It combines music-specific conditioning with an augmented dataset containing beats, chords, key, and tempo descriptions.
- Motivation: Text-to-music systems have improved, but control over tempo, key, chord progressions, and musicality remains limited.Existing models can match instrumentation and vibe while only partially addressing musically meaningful harmonies and steady tempo.
- Motivation: Limited availability of paired music and textual-description datasets constrains training data for controllable generation.
- Dataset and approach: MusicBench enriches existing descriptions with beats, downbeats, chord progressions, key, and tempo extracted using music information retrieval methods.These attributes are intended to provide control information during generation.
- Dataset and approach: Mustango uses MuNet to guide diffusion-based generation toward tempo, key, chords, beats, and general textual descriptions.
- Dataset and approach: MusicBench contains approximately 53K music-audio and description pairs created by altering harmony, tempo, and volume and enriching captions with musical attributes.
- Reported outcome: Experiments report high-quality music faithful to input text, chords, and beats.
2 Dataset Creation
The paper constructs MusicBench by extracting musical features, converting them into control sentences, and diversifying audio and captions. Mustango then conditions latent diffusion on text, beats, and chords through MuNet cross-attention.
- Feature extraction: MusicBench extracts beats, downbeats, chords, keys, and tempo to enhance text prompts and guide generation.BeatNet estimates beat features, tempo is measured in BPM, and Chordino extracts chord features.
- Feature extraction: Extracted features become control sentences appended to original prompts, such as key, tempo, beat count, and chord progression descriptions.
- Audio and text augmentation: Caption robustness is increased by randomly discarding one to four music-feature sentences and rephrasing prompts for variety.
- Dataset construction: MusicCaps supplies 5,479 usable ten-second clips with English captions, from which TrainA, TestA, TrainB, TestB, and TrainC are formed.TrainB and TestB add four control sentences, while TrainC rephrases TrainB prompts.
- Mustango architecture: Mustango uses a latent diffusion model to reduce computational complexity while retaining diffusion expressiveness.A VAE produces latent audio codes, and reverse diffusion reconstructs audio from Gaussian noise under conditioning.
- Mustango architecture: MuNet conditions denoising on text, beats, and chords through additional cross-attention layers, prioritizing beat conditioning before chords.Beat and chord encoders combine music representations with positional or chord embeddings.
- Mustango architecture: Additional cross-attention layers let MuNet integrate music conditions during denoising, whereas Tango uses one layer for text conditions only.The paper links this design to more controllable and meaningful music generation.
4 Experiments
The experiments evaluate Mustango’s audio quality and controllability against Tango, MusicGen, and AudioLDM2 using objective metrics and listening studies. Results show strong music quality and especially improved control over musical attributes, while dataset distribution and model-scale differences affect comparisons.
- Evaluation setup: The evaluation combines objective audio-quality and controllability metrics with general-audience and expert listening studies.The experiments use TestA, TestB, FMACaps, and multiple listening-test rounds.
- Audio quality: Mustango outperforms MusicGen and AudioLDM2 in FAD and KL across TestA, TestB, and FMACaps.Mustango trained from scratch is also comparable in FD and KL to pretrained Mustango and Tango trained on MusicBench.
- Evaluation caveats: Comparisons are constrained because Mustango uses MusicBench-like data distributions, whereas MusicGen and AudioLDM2 use broader large-scale training data.The authors note that MusicGen and AudioLDM2 improve more on FMACaps than on TestA and TestB, and their reported MusicCaps results differ from their original papers.
- Controllability: MusicBench training improves key controllability, while Mustango achieves the strongest chord controllability and performs well on beat and tempo metrics.On FMACaps, Mustango’s CMOT reaches 75.83; overall, it excels particularly in key and chord control.
- Subjective quality: Listening studies find MusicBench-trained models outperform the Tango baseline, with scratch-trained Mustango leading audio quality, rhythm presence, and harmony ratings.The first study included 48 participants, including 26 with more than five years of formal musical training.
- Subjective controllability: Expert listening confirms stronger chord matching for Mustango, while comparison with MusicGen shows higher relevance but similar overall music-quality ratings and weaker rhythmic consistency.Mustango dominates MCM; MusicGen outperforms Mustango in RC.
- Ablation findings: Pretraining on broad audio-text data does not benefit music generation in the reported experiments, although the checkpoints may remain useful for music with soundscapes.The comparison used a pretrained Tango checkpoint initialized from 1.2 million text-audio pairs.
- Ablation findings: MuNet significantly improves controllability and several objective metrics without compromising quality when control sentences are absent.The authors report improvements under both objective and subjective evaluations, while TestA performance indicates control predictors do not reduce performance relative to Tango.
5 Conclusion
Mustango is presented as a controllable diffusion-based text-to-music system that follows music properties embedded in user-specified prompts. Its MuNet module enables greater controllability than the compared text-to-music systems.
- Mustango generates music that follows specified music properties embedded within user-specified text prompts.
- MuNet enables greater music controllability over Tango, AudioLDM2, and MusicGen.
6 Limitations
The method's controllability is limited to Western music because its control information may be absent or differently represented in other musical traditions. It also assumes paired captions and currently generates music up to ten seconds long.
- Controllability is limited to Western music because chords and keys may be missing or represented differently in non-Western music.
- Mustango assumes paired text captions for training and is limited to generating music up to ten seconds because of computational constraints.
- Generating long-form music is left for future work.
7 Ethical Considerations
The study uses MusicCaps-derived music samples and performs its custom augmentation strategies solely on that dataset. Human listening-test data collection was approved by an independent ethics review board.
- The training data is based on 5.5k music samples from MusicCaps, sourced from YouTube.
- The custom data augmentation strategies were performed solely on the MusicCaps dataset, without privately licensed datasets.
- Listening-test data collection involving human annotators was approved by an independent ethics review board.
A Reverse Diffusion Process
The reverse process iteratively reconstructs z0 by modeling each preceding latent state conditioned on the current state and conditions. During inference, guidance scale w is used, while training directly estimates noise with randomly dropped conditions.
- The reverse process iteratively reconstructs z0 from successive latent states.
- Inference uses guidance scale w, whereas training directly uses the noise estimate with conditions randomly dropped.
B Training Details
Mustango uses dropout strategies during training to improve robustness to missing, incomplete, or faulty conditioning inputs. The training setup also includes substantial computational resources and multi-day runs.
- Conditioning robustness: Three dropout strategies remove all inputs, individual features, or portions of prompts during training.The first two target incomplete or missing information, while prompt masking targets robustness to short text inputs.
- Baseline training: 10% text dropout replaces Mustango’s first two feature dropouts when training the Tango baseline.Tango lacks music-feature inputs, so its adaptation drops all text with a single 10% probability.
- Compute: 4 Nvidia Tesla V100 and 8 Quadro RTX 8000 GPUs supported training runs lasting 5 to 10 days with effective batch size 32.
C Performance of the Predictors
The predictor analyses examine whether Mustango follows chord controls and whether its control conditioning preserves performance without explicit control sentences. Results show frequent chord adherence, stronger control metrics with explicit controls, and patterned predictions when controls are absent.
- No-control prompts: Mustango outperforms Tango across most TestA metrics, indicating that control predictors do not compromise performance without explicit control sentences.
- Example comparisons: With control sentences, a Bbm–Ab prompt produced predicted chords Bbm and Ab, while extracted audio chords were F#maj7 and Ab.
- Chord adherence: Mustango generally follows predicted chords, with musically close substitutions such as G to G6, C to Cmaj7, and C#m7 to C#m.A Bbm-for-F#maj7 substitution is described as understandable musically but invalid under the proposed chord-control metrics.
- No-control prompts: Without explicit control sentences, chord predictions often repeat one chord or alternate between two chords.The paper identifies patterns such as A, B, A, B, A, B and A, A, A, A, A, A.
D Insights from the Human Annotation
The qualitative examples compare Tango and Mustango through spectrograms and chord analyses, illustrating more consistent rhythm and closer alignment between Mustango’s predicted and generated chord sequences. The section also situates these observations among related text-to-music systems.
- Rhythm: Mustango’s blues example shows consistent rhythm, whereas pre-trained Tango fine-tuned on MusicBench begins more abruptly.The authors relate this contrast to the expert listening-study results.
- Chord alignment: Mustango’s generated chord sequence and onset timing closely match the predicted G7, F7, C7, G7 conditions.The extracted audio contains G7, F7, C, G7 at nearby times; the C7-to-C difference is described as a minor possible generation or extraction error.
- Related models: Table 4 provides a high-level comparison among recent text-to-music models.
- Related models: Prior systems span semantic-token, latent-diffusion, autoregressive, and diffusion-based approaches for text-to-audio or text-to-music generation.The cited examples include AudioLM, AudioLDM, Tango, MusicLM, Noise2Music, Ernie-Music, and MusicGen.
F FMACaps dataset creation
FMACaps is an evaluation set built from 1,000 Free Music Archive clips, enriched with expert- and model-generated descriptions, extracted tags, and music-specific control sentences. The dataset supports evaluation with varied numbers of explicit controls and documents the templates used to express musical attributes.
- Source construction: 1,000 random FMA-large songs were clipped into 10-second fragments and tagged for mood, genre, instrumentation, voice, and voice gender.A music expert described 25 samples, while ChatGPT generated pseudo-prompts for the remainder from extracted tags.
- Prompt design: FMACaps prompts received 0, 1, 2, 3, or 4 control sentences with probabilities of 25%, 30%, 20%, 15%, and 10%, respectively.Control sentences were added after extracting relevant music features.
- Evaluation interface: The listening-test interfaces included separate general and music-expert controllability question formats.
- Caption rephrasing: ChatGPT rephrasing converts prompts into a single natural paragraph using the supplied musical terms, with a maximum of eight sentences.The instruction excludes the word “modification” and requests only the caption as output.
- Control templates: Control-sentence templates encode detected key, chords, beats, and tempo, with tempo labels assigned by BPM bins from Grave through Prestissimo.Examples include statements specifying a song’s key, tempo, beat count, and chord progression.
J Custom Captions used for Listening Studies
The listening studies used custom captions, including general and opposing prompts, to compare samples generated by five text-to-music systems. Additional examples examine whether generated music follows specified chords, beats, tempo, and dynamic changes.
- The general listening test used custom captions, with captions in the top part reused across both runs and bottom-part captions added only in the second run.
- The control experiment used 20 expert-written prompts arranged as 10 realistic contrasting pairs without contradicting elements.Each pair shared an original caption while contrasting specific musical instructions.
- Samples from pre-trained Tango, Tango fine-tuned on MusicBench, Mustango, MusicGen-M, and AudioLDM2 were generated from the same prompts.
- Chord control: Mustango’s reggae sample matches the prompted chord sequence, whereas the MusicBench-fine-tuned Tango sample contains too many chords and the MusicCaps-fine-tuned sample is noisy and dissonant.The Mustango example uses a prompt specifying Am, Dm, G, and C chords at Adagio pace.
- Beat and arrangement control: For the metal prompt, Mustango produces four distinguishable opening drum beats followed by guitars, while the Tango variants show noise from the start or some initial noise.The prompt specifies Em, C, G, and D chords and a tempo of 120 bpm.
- Dynamic control: Mustango makes the classical sample’s gradual volume increase apparent, unlike MusicGen-M, where it is unclear, and AudioLDM-2, where it is absent.The classical prompt describes a melancholic A-minor piece with the chord sequence Am, C, Am.