Source-linked AI summary
Extending Music Annotation Schemas: Zero-Shot Prediction or Few-Shot Adaptation?
Christos Plachouras, Emmanouil Benetos, Johan Pauwels
TL;DR
Changing catalog needs make schema extension important, while expert annotation is expensive and the best approach across annotation budgets is unclear. This paper benchmarks zero-shot prediction, frozen-representation learning, and model adaptation on MGPHot. Across two extensions, a simple baseline outperformed CLAP and Music Flamingo; frozen-feature heads improved predictions with few annotations, while deeper updates brought smaller, less consistent benefits.
Problem
Expert annotation is costly as music schemas evolve, leaving open how to extend existing catalogues and when supervised approaches outperform zero-shot audio-language prediction.
Source: §1 · p. 1Method
The paper benchmarks MGPHot schema extension across annotation budgets, comparing zero-shot audio-language prediction, frozen representation reuse, and adaptation of models trained on existing attributes.
Source: §1 · p. 1Results
Across two MGPHot extensions, even a simple baseline outperformed CLAP and Music Flamingo; frozen-feature heads improved predictions with few annotations, while deeper CNN updates had smaller, less consistent benefits.
Source: §5 · p. 4Takeaways & Limitations
Reusing existing audio representations with limited new annotation is a practical approach to schema extension, while audio-language models need further research for rating prediction.
Source: §5 · p. 4
Abstract
from arXiv · showhide
Automatic music annotation is typically tackled under the assumption of a fixed annotation schema. In practice, commercial music catalogs often need to accommodate new musical attributes as needs evolve. Given that expert music annotation is expensive, it is not evident which methodological approach is most effective at accommodating new attributes and backfilling existing tracks; audio-language models promise zero-shot prediction, but at what annotation budget does supervised adaptation become more compelling? We propose a benchmark based on the MGPHot popular music annotation dataset for simulating music schema extension across different annotation budgets. We investigate zero-shot prediction with audio-language models, learning new attributes from pretrained representations, and adapting models trained on existing annotations. Our results suggest that supervised adaptation is more effective than zero-shot prediction even with small annotation budgets, while frozen representation reuse remains the most effective approach for modest budgets without the tuning required by deeper adaptation.
1. INTRODUCTION
Music schemas evolve as musical styles and service needs change, but expert annotation is costly. The paper benchmarks ways to add new attributes and predict their values across an existing catalogue.
1. INTRODUCTION
- Expert annotation takes several minutes per track, while changing needs can prompt schema updates and create the problem of extending attributes across an existing catalogue.The schema includes attributes, definitions, and allowed values.Source: §1 · p. 1
- The benchmark compares adapting existing models, training new heads on frozen features, and zero-shot audio-language prediction from attribute descriptions.Adaptation can overfit with limited examples or cause forgetting; frozen representations may lack information for new attributes, while larger-audio pretraining may offer more general features.Source: §1 · p. 1
- Each MGPHot schema extension adds seven attributes to an existing set of 51.Source: Figure 1 · p. 1
- The benchmark compares these method families using expert MGPHot ratings at budgets of 8, 32, and 128 support tracks.The authors describe it as the first benchmark of music schema extension in an industry-motivated setting across contrasting method families.Source: §1 · p. 1
- The contributions include a reusable limited-annotation protocol and comparisons of prediction methods, budget effects, retention of existing attributes, and possible pretraining exposure to test recordings.Source: §1 · p. 1
2. BACKGROUND
Prior work has developed music tagging datasets and transferable audio representations, alongside approaches to learning new tasks or classes. The performance of audio-language models for numerical ratings under stricter comparisons remains understudied.
2. BACKGROUND
- MGPHot introduced expert ratings for 58 musical attributes, with defined scales and quality checks that support evaluation of attribute strength beyond binary presence.Source: §2 · p. 1
- Continual learning updates models for new tasks while seeking to avoid degradation on previous ones, whereas metric learning defines new classes from few examples without retraining for each extension.The cited music metric-learning applications concern classification or tagging; the passage notes no proposed extension to regression tasks.Source: §2 · p. 2
- How audio-language models perform on numerical ratings under stricter conditions and against other approaches remains understudied, with possible pretraining data contamination motivating overlap checks.Source: §2 · p. 2
3.1. Data and schemas
The study uses MGPHot’s continuous expert ratings to construct two schema extensions, then evaluates target-attribute prediction across three support-set sizes.
3.1. Data and schemas
- MGPHot provides continuous ratings from 0 to 1 for 58 attributes across 21,320 tracks, spanning seven musical categories.Source: §3.1 · p. 2
- Each extension adds seven targets to 51 base attributes: E1 holds out all vocal attributes, while E2 selects one attribute from each category.The extensions represent lower and higher expected semantic relatedness to existing attributes, respectively, using category membership as a proxy.Source: §3.1 · p. 2
- Support sets contain 8, 32, or 128 tracks; eight random draws from the artist-disjoint training split are evaluated on test recordings using macro RMSE over seven targets.The canonical split contains 12,800 training tracks and 4,286 test tracks; results report the mean and sample standard deviation across draws.Source: §3.1 · p. 2
3.2. Validation
Because small support sets leave too few examples for reliable validation, the study selects hyperparameters using shared base attributes and simulates adaptation on held-out targets.
3.2. Validation
- Rather than further reduce small support sets for validation, the study selects hyperparameters on 45 attributes shared by the E1 and E2 base schemas and their validation set.The selected setup is then applied to target attributes.Source: §3.2 · p. 2
- For deeper adaptation, base models are trained on 38 attributes and adapted to seven held-out attributes, simulating category-held-out and cross-category extensions.Source: §3.2 · p. 2
3.3. Methods
The methods compare zero-shot audio-language prediction, regression on CLAP similarities, frozen audio-feature heads, and deeper CNN adaptation under shared annotation budgets, with no-audio and fully supervised references.
3.3. Methods
- The methods compare zero-shot prediction, supervised feature reuse and adaptation, and no-audio and fully supervised references.The approaches are evaluated under shared experimental conditions.Source: §3.3 · p. 2
3.3.1. CLAP zero-shot
- CLAP zero-shot ratings use text anchors for attribute ranges, weight their values by audio-text similarity, and average predictions across recording windows.Validation selected five anchors including negatives, music-speech-AudioSet, and T = 0.07.Source: §3.3.1 · p. 2
3.3.2. Music Flamingo zero-shot
- Music Flamingo is evaluated with numeric-rating prompts and anchor-option prompts; validation selects its five-anchor option-based formulation.That formulation slightly improves RMSE over definition-only numeric prediction.Source: §3.3.2 · p. 3
3.3.3. CLAP similarity regression
- CLAP similarity regression fits a linear model to support-track similarities, with validation selecting the high-minus-low endpoint difference as input.Using all five anchor similarities improves validation error at 128 support tracks but worsens it at eight and 32.Source: §3.3.3 · p. 3
3.3.4. Frozen audio heads
- Frozen CLAP or CNN audio features feed one ridge-regression head per attribute, avoiding forgetting because the encoders remain fixed.The CNN is pretrained separately for each extension on the 51 base attributes; ridge strength is selected by validation.Source: §3.3.4 · p. 3
3.3.5. CNN adaptation
- CNN adaptation updates selected convolutional blocks, the shared dense layer, and target heads, while retaining fixed base heads and training on base labels.The selected configuration uses 256 replay tracks and adapts the final block; selection balances base and target validation RMSE.Source: §3.3.5 · pp. 3–4
3.3.6. Baselines and reference
- The support-mean baseline predicts each attribute using its support-track mean, while the fully supervised CNN trains jointly on all 58 attributes.The CNN provides a reference for performance using the full training set.Source: §3.3.6 · p. 4
4. RESULTS
Across the two extensions, frozen audio-feature heads improve on CLAP similarity regression, while deeper CNN updates yield smaller, mixed changes; excluding possible pretraining matches shows no clear differential effect.
4. RESULTS
- Both frozen encoders outperform CLAP similarity regression at every budget; CLAP is better for E1 and the CNN for E2, and both approach the fully supervised reference at 128 tracks.The support-track mean also has lower average error than zero-shot methods in both extensions.Source: §4 · p. 4
- Deeper CNN updates produce small, mixed changes versus frozen heads: both eight-track comparisons worsen, while the largest mean improvement is 0.0046 RMSE for E2 at 32 tracks.Base-attribute RMSE changes range from −0.000186 to +0.001028, indicating that existing predictions are largely retained.Source: §4 · p. 4
- After excluding possible pretraining matches, MF target RMSE changes by +0.0004 for E1 and −0.0008 for E2; paired intervals show no clear differential effect versus the fully supervised CNN.Across all attributes, the difference in RMSE changes is +0.0003 (95% interval: [−0.0005, +0.0012]).Source: §4 · p. 4, Figure 3 · p. 4
5. CONCLUSION
The paper introduces a public benchmark for music schema extension and compares zero-shot prediction with supervised adaptation under common annotation budgets.
5. CONCLUSION
- On two MGPHot extensions, even a simple baseline outperforms CLAP and Music Flamingo; frozen-feature heads improve further with a few annotations.A CNN trained on existing attributes remains competitive with pretrained CLAP representations.Source: §5 · p. 4
- Updating the CNN more deeply brings smaller, less consistent benefits than reusing frozen representations.The authors identify representation reuse with limited new annotation as a practical approach to schema extension.Source: §5 · p. 4
- The benchmark is intended to support further testing of the trade-off between annotation cost and prediction accuracy in music.The paper also calls for research improving audio-language models for non-conversational tasks such as rating prediction.Source: §5 · p. 4
6. COMPLIANCE WITH ETHICAL STANDARDS
The study analyzes existing MGPHot music recordings and annotations; no ethical approval was required.
6. COMPLIANCE WITH ETHICAL STANDARDS
- The study analyzes existing MGPHot recordings and annotations and required no ethical approval.Source: §6 · p. 4