Source-linked AI summary
AllMusicCaps: Album Reviews as Complementary Supervision for Music CLAP
Pablo Alonso-Jiménez, Xavier Lizarraga-Seijas, Xavier Serra, Dmitry Bogdanov
TL;DR
Existing music CLAPs rely largely on accurate but expressively narrow tag- and web-derived captions, leaving richer narrative and evaluative descriptions underrepresented. The paper converts expert AllMusic reviews into AllMusicCaps and revises CLAP training with layer selection and SigReg. Combined review supervision improves complex-query retrieval, while the resulting recipe leads across most reported retrieval, classification, and probing evaluations.
Problem
Tag- and web-derived captions are expressively narrow, leaving narrative, evaluative, and figurative music descriptions underrepresented.
Method
The paper distills expert AllMusic reviews into AllMusicCaps and combines review captions with existing corpora while testing encoder-layer selection and SigReg-regularized training.
Results
Combined review supervision produces the largest Song Describer gains for high-complexity captions, and the resulting model leads on text-to-music retrieval, MLP probing, and zero-shot classification except MTG-Jamendo Instrument.
Takeaways & Limitations
Album reviews function as a complementary supervision source for complex music-text retrieval, while SigReg improves representation learning under the reported conditions.
Abstract
from arXiv · showhide
Recent open text-audio contrastive models (CLAPs) are typically trained with LLM-generated captions derived from tag datasets or web search results, which tend to be accurate but expressively narrow. As a complementary source, we explore human-written album reviews, specifically expert reviews from AllMusic: they exist at scale and carry narrative cues, evaluative adjectives, and scene framing that other sources lack. Since raw reviews are too noisy for direct use as captions, we first build a caption corpus with 24,5346 samples via an LLM preprocessing pipeline that identifies descriptive musical quotes and rewrites them into training-ready captions. We find that album review supervision yields the largest retrieval gains on a human-written caption benchmark (Song Describer), particularly for complex queries that other existing caption datasets leave uncovered. In addition, we revisit the training recipe and show that SigReg regularization, which encourages an isotropic Gaussian distribution in the embedding space, improves MLP probing across classification tasks, as well as text-to-music retrieval. The resulting model outperforms open CLAP-style baselines on text-to-music retrieval, zero-shot classification, and most MLP probing tasks. We release the review-derived caption dataset and model weights to support future research.
1. INTRODUCTION
The paper uses expert AllMusic album reviews to add narrative and evaluative musical descriptions missing from conventional caption sources. It introduces AllMusicCaps and combines review supervision with revised training objectives to improve music-text representation learning.
- Motivation: Conventional LLM-generated captions are accurate and objective but omit narrative framing and evaluative musical descriptions.Examples include “feels like...” queries and descriptions such as “raspy vocals.”
- Approach: AllMusic reviews provide expert-written, large-scale supervision beyond tag-like information, but require preprocessing because they include multi-track and non-audio commentary.The paper compares two LLM caption-generation strategies to make reviews suitable for training.
- Contributions: AllMusicCaps is a review-derived music caption dataset enriched with YouTube and Discogs metadata for multimodal text-audio research.The dataset and associated materials are released for non-commercial scientific research.
- Contributions: The paper studies audio-encoder layer selection and training objectives, consolidating the findings into a recipe that improves retrieval, zero-shot classification, and probing.The recipe includes combining encoder layers and revisiting the training objective.
- Contributions: SigReg regularization benefits text-to-music retrieval and probing under certain conditions.SigReg is adapted from LeJEPA and encourages isotropic Gaussian representations.
2. RELATED WORK
Prior music CLAP systems mainly use tags, playlists, professional captions, or web-retrieved text. This work distinguishes itself by using expert album-review text at scale and combining SigReg with contrastive training.
- Music text-audio supervision: Existing music CLAPs draw supervision from tag datasets, private professional captions, weak descriptions, playlists, or music tagging corpora.Album-review text had not previously been explicitly used at scale across these systems.
- Caption generation: LP-MusicCaps converts tag sets into natural-language captions, while later approaches refine prompting or retrieve web information.AllMusicCaps instead uses expert-written album reviews as its underlying source.
- Self-supervised representation learning: Most joint-embedding approaches use contrastive or self-supervised objectives, including InfoNCE, BYOL adaptations, and masked token modeling.The paper combines LeJEPA’s SigReg regularizer with InfoNCE to retain probing benefits without the retrieval cost of negative-free training.
3. DATA
The data pipeline links AllMusic reviews with Discogs and YouTube metadata, then generates captions through quote-based or structured LLM pipelines. The resulting common corpus contains 245 346 tracks and can be combined with existing caption datasets.
- Data sources: The source collection contains 94 041 AllMusic album reviews paired with Discogs release metadata by artist and album matching.YouTube links associated with the matched Discogs releases provide corresponding audio clips and metadata.
- Data sources: The matching process yields 245 346 audio tracks paired with album-level reviews and Discogs information plus track-level YouTube metadata.Because most review text does not directly describe audio, two distillation pipelines are introduced.
- AMCQuotes: AMCQuotes extracts factual album- and song-level musical descriptions, then rewrites track-specific captions using review quotes and metadata.The pipeline prioritizes YouTube tags and matched song quotes when these conflict with album-level metadata, with outputs shorter than 15 words.
- AMCStruct: AMCStruct generates dense captions in one pass from reviews, YouTube data, and Discogs tags using fields for style, mood, energy, tempo, instrumentation, and production.It maximizes coverage but carries a higher hallucination risk than the quote-based pipeline.
- Dataset release and use: Both pipelines produce validated captions for a common subset of 245 346 tracks, which is released as AllMusicCaps and can complement existing open caption datasets.The combined training setup considers LP-MusicCaps and M4-RAG alongside the review-derived captions.
4. METHOD
The model uses a two-tower CLAP architecture with staged contrastive training, then evaluates review-derived and baseline combinations across retrieval, classification, similarity, and probing protocols.
- 4.1 Architecture: The two-tower CLAP projects averaged 10-second audio and encoded text into a shared 512-dimensional embedding space.The audio encoder is OMAR-RQ small, while the text encoder is all-MPNet-base-v2.
- 4.2 Pre-training with raw metadata: A raw-metadata pre-training stage uses 6.5 M audio tracks paired with YouTube and Discogs metadata without LLM-generated captions.The stage pre-aligns the towers before captioned training.
- 4.2 Pre-training with raw metadata: The contrastive objective symmetrizes audio-to-text and text-to-audio losses over normalized embeddings in each batch.The formulation uses temperature-scaled similarities, with the reverse-direction loss defined symmetrically.
- 4.3 Training: Captioned training initializes from the pre-trained checkpoint and uses a 150 k-step schedule with an effective batch size of 3072.Models use AdamW, cosine annealing, and 15 k warm-up steps, with corpus sampling ratios held across experiments unless stated otherwise.
- 4.3 Training: Table 1 compares individual review-derived corpora, the LPMC + M4-RAG + FS + PSE baseline, and their combinations.The table reports downstream performance across the evaluation metrics, with the best result bolded per column.
- 4.4 Evaluation: The study evaluates models on five downstream protocols spanning retrieval, classification, similarity, and probing.Retrieval reports MRR on MusicCaps and Song Describer; zero-shot classification reports accuracy on GTZAN and FMA-Small.
- 4.4 Evaluation: Table 2 ranks the top three queries by improvement from review supervision, using ∆rank relative to the baseline.Larger ∆rank indicates a larger rank improvement for the review-augmented model.
5. EXPERIMENTS
The experiments show that review-derived captions complement existing corpora by improving retrieval mainly for complex, narrative queries, while learned all-layer audio representations and SigReg improve downstream performance under specific conditions.
- Training data analysis: 2.8 and 3.7 Song Describer MRR points result from adding AMCStruct and AMCQuotes, respectively, to the baseline corpus.Reviews alone underperform the four-corpus baseline on every metric, whereas MusicCaps and zero-shot classification change only marginally after augmentation.
- Training data analysis: AMCQuotes consistently outperforms AMCStruct when added to existing corpora, although prompting strategy and LLM vary between the two pipelines.The authors attribute the difference to AMCQuotes’ more narrative register but note that the factors cannot be fully separated.
- Training data analysis: Review supervision improves Song Describer MRR at every complexity level, with the largest gains on score 3 and 4 captions.MusicCaps MRR remains essentially flat across complexity levels.
- Training data analysis: 46.5% of score ≥3 Song Describer captions contain review-style terms versus 3.0% recording-style terms, while MusicCaps shows the reverse pattern at 28.5% and 46.8%.The analysis links the retrieval asymmetry to narrative, evaluative, and scene-framing language concentrated in Song Describer.
- Audio encoder layer selection: A learned weighted sum of all OMAR-RQ layers improves MusicCaps and Song Describer MRR by 0.5 absolute and DimSim accuracy by 8.7 percentage points over the layer-12 baseline.The comparison includes final-layer-only, layer-6, and learned all-layer configurations.
- Training objective: InfoNCE+SigReg and LeJEPA consistently beat InfoNCE on classification probing, while LeJEPA’s lack of negatives penalizes every non-probing task.Most models peak at 20–60k steps and can degrade on non-probing tasks despite continuing train/validation loss decreases.
- Comparison with external baselines: Across most configurations and training steps, the authors outperform the strongest external baseline on MusicCaps, GTZAN, FMA-Small, Genre, Mood, and MGPHot.No model surpasses the strongest baseline on MTG-Jamendo Instrument, and only InfoNCE+SigReg does so on MTT.
6. CONCLUSIONS
The paper introduces AllMusicCaps and combines it with existing datasets and training changes to improve CLAP performance across several evaluations. It releases the dataset, model weights, and code for non-commercial research.
- 6. CONCLUSIONS: 245 346 captions form AllMusicCaps, distilled from AllMusic album reviews and supplementary metadata.The dataset is released for multimodal research in music.
- 6. CONCLUSIONS: The resulting CLAP model leads on text-to-music retrieval, MLP classification, and zero-shot classification, except for MTG-Jamendo Instrument.The recipe combines existing datasets, weighted audio-encoder layers, and SigReg regularization.
- 6. CONCLUSIONS: AllMusicCaps, model weights, and code are released for non-commercial research purposes.The paper identifies deeper study of geometrical regularization and music captioning as future work.
8. AI USAGE STATEMENT
LLMs support both caption creation and annotation in the study, while autocomplete and agentic coding tools assist implementation and LLMs help revise the paper's writing.
- 8. AI USAGE STATEMENT: Qwen 2.5-32B generates AMCQuotes captions, while Llama 3-70B generates AMCStruct captions and caption-complexity annotations.These uses correspond to Sections 3 and 5.1.
- 8. AI USAGE STATEMENT: Autocomplete and agentic coding tools assist the implementation of the experiments.
- 8. AI USAGE STATEMENT: LLMs review and improve the paper's writing in addition to their methodological use.