Source-linked AI summary
Music Understanding LLaMA: Advancing Text-to-Music Generation with Question Answering and Captioning
Shansong Liu, Atin Sakkeer Hussain, Chenshuo Sun, Ying Shan
TL;DR
Public music-caption datasets are scarce, and existing audio question-answering resources are not well suited to open-ended music questions. The paper builds MU-LLaMA and MusicQA from public captioning and tagging data; MU-LLaMA outperforms reported state-of-the-art models in both music question answering and captioning.
Problem
Publicly available music-caption data is scarce, and existing audio question-answering datasets and models lack suitable depth or music-focused training for open-ended music questions.
Source: §1 · p. 1Method
The authors train MU-LLaMA with MERT music representations and MusicQA question-answer pairs generated from public music captions and tags.
Source: §1 · p. 2, §3 · pp. 2–3Results
MU-LLaMA surpasses current state-of-the-art models in music question answering and captioning; in question answering, it exceeds LTU by more than 10% absolute in METEOR and ROUGEL.
Source: §6 · p. 4, §5.2 · p. 4Takeaways & Limitations
Abstract
from arXiv · showhide
Text-to-music generation (T2M-Gen) faces a major obstacle due to the scarcity of large-scale publicly available music datasets with natural language captions. To address this, we propose the Music Understanding LLaMA (MU-LLaMA), capable of answering music-related questions and generating captions for music files. Our model utilizes audio representations from a pretrained MERT model to extract music features. However, obtaining a suitable dataset for training the MU-LLaMA model remains challenging, as existing publicly accessible audio question answering datasets lack the necessary depth for open-ended music question answering. To fill this gap, we present a methodology for generating question-answer pairs from existing audio captioning datasets and introduce the MusicQA Dataset designed for answering open-ended music-related questions. The experiments demonstrate that the proposed MU-LLaMA model, trained on our designed MusicQA dataset, achieves outstanding performance in both music question answering and music caption generation across various metrics, outperforming current state-of-the-art (SOTA) models in both fields and offering a promising advancement in the T2M-Gen research field.
1. INTRODUCTION
Public music-caption data is scarce, and volunteer annotation is too slow and unpredictable to produce the scale of text–music pairs needed for text-to-music research. The paper introduces MU-LLaMA and a MusicQA dataset built from public music captions and tags, targeting music question answering and caption generation.
1. INTRODUCTION
- MusicCaps, the largest publicly available dataset cited for music captions, contains approximately 28.52 hours, while many captioned music datasets are closed source because of licensing restrictions.The paper argues that publicly usable text–music pairs are needed for music generation research.Source: §1 · p. 1
- Crowdsourced annotation through Song Describer is time-consuming and uncontrollable, so the authors propose using an LLM to caption large numbers of public music files.Song Describer recruits volunteers through an online platform to annotate provided music.Source: §1 · p. 1
- Existing audio question-answering models have limited suitability for music: LLaMA Adapter and UniVAL lack music-text pretraining, while LTU mainly uses regular audio and its code, model, and dataset were unreleased.The cited comparison motivates a music-focused question-answering model and dataset.Source: §1 · p. 1
- MU-LLaMA performs music question answering and captioning, with reported superior performance across various metrics over SOTA models.The paper also proposes a systematic way to create music question-answering data and demonstrates caption generation in formats for text-to-music models.Source: §1 · p. 2
2. MUSIC FEATURE REPRESENTATION
The paper compares pretrained audio representations on music tagging in MagnaTagATune to select an encoder for MU-LLaMA. MERT performs best on the reported tagging metrics and is selected for music representation.
2. MUSIC FEATURE REPRESENTATION
- The representation comparison evaluates pretrained audio models on music tagging in MagnaTagATune to identify an encoder for MU-LLaMA.The candidate representations include ImageBind, JukeBox, OpenL3, CLAP, Wav2CLIP, and MERT.Source: §2 · p. 2
- The candidate encoders use different representation strategies: cross-modal embeddings, generative music modeling, audio-visual learning, contrastive audio-text pretraining, and distilled CLIP embeddings.MERT is described as an acoustic music representation model trained with large-scale self-supervision and acoustic and musical teachers.Source: §2 · p. 2
- MERT performs best on MTTAUC and MTTAP in music tagging on MagnaTagATune, so the authors select it to represent music for MU-LLaMA.MTTAUC denotes predicted-tag accuracy; MTTAP denotes average precision of predicted tags.Source: Table 1 · p. 2, §2 · p. 2
3. MUSICQA DATASET GENERATION
The paper uses MPT-7B to create music question-answer pairs from existing captioning and tagging datasets. Its instructions generate both descriptive responses and open-ended questions about musical attributes.
3. MUSICQA DATASET GENERATION
- MPT-7B generates music question-answer pairs from existing music captions or tags because public music datasets typically lack ready-made pairs.The authors devise instructions that guide the model to generate responses from the source text.Source: §3 · p. 2
- One instruction set elicits answers to broad prompts such as describing the music, what is heard, and what can be inferred from the audio.The prompts use an input caption or list of tags.Source: §3 · p. 2
- A second instruction set generates five open-ended pairs about attributes such as emotion, tempo, genre, instruments, and overall tone.MusicCaps descriptions prompt paraphrased questions and answers; MPT infers suitable pairs from MagnaTagATune tags.Source: §3 · pp. 2–3
4. MU-LLAMA MODEL ARCHITECTURE
MU-LLaMA combines Meta’s LLaMA with MERT music features through a music understanding adapter. The adapter’s final embedding supplies music context to LLaMA, while the MERT encoder and LLaMA Transformer layers remain frozen during adapter finetuning.
4. MU-LLAMA MODEL ARCHITECTURE
- MERT music features are projected and processed by a three-sub-block dense adapter before supplying music context to LLaMA’s final layers.The final embedding has 4096 dimensions and is used in the last L−1 LLaMA layers.Source: §4 · p. 3
- Each adapter sub-block combines normalization, a linear layer, and SiLU activation, with a skip connection carrying the previous layer’s input forward.Source: Figure 2 · p. 3, §4 · p. 3
- During training, the MERT encoder and LLaMA Transformer layers are frozen while the music understanding adapter is finetuned.Source: §4 · p. 3
5. EXPERIMENTS AND RESULTS
Experiments evaluate MU-LLaMA on music question answering and captioning using MTG-Jamendo audio. The reported results place it ahead of the compared models on both tasks.
5.1. Experiment Setup
- Evaluation used MTG-Jamendo tracks with no overlap with MusicQA training audios; 4,500 QA pairs came from 500 tracks, while MPT generated captions for 1,000 tracks.Source: §5.1 · p. 4
5.2. Music Question Answering
- MU-LLaMA exceeds LTU by more than 10% absolute and LLaMA Adapter by more than 5% on METEOR and ROUGEL.Source: §5.2 · p. 4
5.3. Music Captioning
- For captioning, MU-LLaMA improves over four compared models on BLEU, METEOR, and ROUGEL; the authors attribute this to pretrained MERT and finetuned LLaMA-2 on MusicQA.Source: §5.3 · p. 4
5.2. Music Question Answering; 5.3. Music Captioning
- MU-LLaMA leads on music question answering and captioning, outperforming two QA baselines across four metrics and topping four captioning models on BLEU, METEOR, and ROUGEL.Source: Table 2 · p. 4, Table 3 · p. 4, §5.2 · p. 4
6. CONCLUSION
The paper introduces MU-LLaMA for music question answering and captioning, alongside a method for constructing music QA datasets from captioning and tagging datasets. It reports superior performance on both tasks and identifies future work on lyric-related speaker comprehension and T2M-Gen dataset curation.
6. CONCLUSION
- MU-LLaMA is designed for music-enabled question answering and captioning, alongside a method for constructing music QA datasets from captioning and tagging datasets.Source: §6 · p. 4
- The authors report that MU-LLaMA surpasses current state-of-the-art models in music question answering and music captioning, with strong generalization capabilities.Source: §6 · p. 4
- Future work is anticipated on speaker comprehension for lyric-related questions and using MU-LLaMA to curate datasets for text-to-music generation.Source: §6 · p. 4