LP-MusicCaps: LLM-Based Pseudo Music Captioning

1. INTRODUCTION

• The only publicly available track-level caption dataset described, MusicCaps, contains just 5,521 music-caption pairs.Other cited datasets were private or difficult to reproduce or afford.Source: §1 · p. 1
• Tag-derived captions can inherit high false-negative rates and text inconsistencies in synonyms, punctuation, and singular or plural forms.Without proper treatment, these issues can limit captioning-model performance.Source: §1 · pp. 1–2
• The paper applies LLMs to tagging datasets to generate captions intended to be tag-consistent, grammatical, and vocabulary-enriched, taking a pragmatic data-level approach.The authors frame this as addressing captioning difficulty through data rather than theory or model design.Source: §1 · p. 2
• Earlier pseudo-caption approaches reported task performance but did not directly evaluate generated-caption quality; this paper proposes a systematic evaluation scheme.Source: §1 · p. 2
• Models trained with LP-MusicCaps perform well in both zero-shot and transfer-learning scenarios, supporting the use of LLM-based pseudo-music captions.Source: §1 · p. 2

2.1 Large Language Model for Data Generation

• GPT-3.5 Turbo generates sentences from multi-label tags and task instructions; metadata is excluded because it could prompt confident but inaccurate captions.The authors also add a hallucination question to their evaluation scheme.Source: §2.1 · p. 2

2.2 Task Instruction Design

• Four prompts cover tag-grounded writing, compression, creative paraphrase, and co-occurring-attribute prediction; “include/with” wording curbs hallucination, and only the description becomes the caption.Source: §2.2 · pp. 2–3

3.1 Objective Evaluation

• Objective evaluation compared generated captions with MusicCaps references using n-gram overlap, BERT-Score semantic similarity, and vocabulary-diversity measures.The diversity measures were treated as subsidiary rather than overall caption-quality measures.Source: §3.1 · p. 3

3.2 Subjective Evaluation

• The human A-vs-B study collected 480 ratings from 24 music researchers or industry professionals evaluating 240 clips against ground-truth captions.Each rater assessed 20 randomly selected questions, judging accurate attributes and incorrect attributes separately.Source: §3.2 · pp. 3–4

3.3 Results

• The proposed method achieved higher BERT-Score while generating diverse vocabulary; tag concatenation and K2C produced incomplete or unrelated captions, while templates benefited from musical context.The authors report that the newly created vocabulary did not harm music semantics.Source: §3.3 · p. 4
• Writing showed high n-gram performance; Summary was shortest and had the highest BERT-Score; Paraphrase expanded vocabulary; Attribute Prediction had lower BLEU but competitive METEOR.Summary also had competitive ROUGE-L; Attribute Prediction's METEOR result reflects credit for semantically similar words.Source: §3.3 · p. 4
• For true-positive ratings, instruction-based methods beat K2C in win-plus-tie scores; Attribute Prediction scored highly, but was slightly more false-positive-prone than ground truth.All methods except Attribute Prediction had more win-plus-tie than loss scores on false positives.Source: Figure 2 · p. 4, §3.3 · p. 5

4. DATASET: LP-MusicCaps

• The dataset uses MusicCaps (5,521 examples), MTT (26k clips, reduced to 22k with tags), and MSD (0.52 million 30-second clips) as tag sources.The authors report using 5,495 MusicCaps samples after losing 26 data samples.Source: §4 · p. 5
• The three source datasets use averages of 10.7 labels per MC clip, 3.3 per MTT clip, and 10.2 per MSD clip to generate captions.MTT's full 188-tag vocabulary was used, and tracks without associated tags were excluded.Source: §4 · p. 5
• LP-MusicCaps-MC has four times more captions per audio than manually written captions, while LP-MusicCaps-MSD has the largest audio duration among music caption datasets discussed.The dataset family is publicly accessible and provided at various scales; MuLaMCap is not publicly accessible.Source: §4 · p. 5

5.1 Encoder-Decoder Model

• The model encodes log-mel spectrograms with convolutional layers and transformer blocks, then uses cross-modal attention to predict caption tokens autoregressively.Source: §5.1 · pp. 5–6

5.3 Results

• LP-MusicCaps pre-training wins BERT-Score and most n-gram metrics in transfer learning, while zero-shot models generally trail the supervised baseline, with few exceptions.Source: Table 4 · p. 6, §5.3 · p. 6

5.2 Experimental Setup

• The evaluation compares supervised MusicCaps training with LP-MusicCaps-MSD pre-training in zero-shot and fine-tuning settings, alongside pseudo-caption baselines.Source: §5.2 · p. 6
• Inputs are 10-second, 16 kHz audio; pre-training uses balanced tag sampling, batch size 256, and 32,768 updates, while supervised and transfer learning use 100 epochs.Source: §5.2 · p. 6

• LLM-based tag-to-pseudo-caption generation addresses music-caption data scarcity and produces the large-scale LP-MusicCaps dataset.Source: §6 · p. 6
• Models trained with LP-MusicCaps showed improved generalization, and the approach has potential to reduce music-language dataset collection time and cost.Source: §6 · p. 6
• Community collaboration and human evaluation are essential to enhance the quality and accuracy of generated captions.Source: §6 · p. 6
• The authors suggest exploring LLMs for other music information retrieval and music recommendation topics.Source: §6 · p. 6