Source-linked AI summary
What Makes a Good Layer? Assessing the Layer-Wise Intrinsic Properties of Music Foundation Models
Angelos-Nikolaos Kanatas, Yuexuan Kong, Pablo Alonso-Jiménez, Xavier Serra, Dmitry Bogdanov
TL;DR
Choosing which layer to extract from music foundation models is largely heuristic. This paper analyzes intrinsic layer properties across models and tasks, finding task- and paradigm-dependent proxies, with pitch-transposition equivariance consistently indicating tonal quality and intrinsic metrics supporting efficient layer selection.
Problem
Layer selection for frozen music foundation model representations remains largely heuristic, limiting principled understanding of transfer quality across depth, tasks, and pre-training paradigms.
Method
The study analyzes 12 music foundation models across three pre-training paradigms using geometric, spectral, invariance, discriminability, and pitch-transposition equivariance properties.
Results
Intrinsic metrics track layer-wise performance on several tasks but fail on tonality, while pitch-transposition equivariance provides a more consistent tonal-quality proxy across model families.
Takeaways & Limitations
Intrinsic metrics can shortlist strong layers and match or outperform trainable multi-layer fusion, particularly when labeled data is scarce.
Takeaways & Limitations
Metric associations are task- and paradigm-dependent, with some reversing across model families and raw values failing to support reliable cross-model rankings.
Abstract
from arXiv · showhide
Music foundation models are commonly used as frozen audio feature extractors, yet selecting which layer to extract from remains largely heuristic. Current practice defaults to fixed depths or multi-layer fusion, with limited understanding of why certain layers transfer better across downstream tasks or how representation quality varies with depth and pre-training paradigm. We conduct a systematic layer-wise analysis of 12 music foundation models spanning three pre-training paradigms (masked modeling, autoregressive modeling, and contrastive learning), characterizing their hidden representations through intrinsic geometric and transformation-based properties. Correlating label-free representation-quality metrics with layer-wise performance across 15 downstream tasks, we find that several metrics track layer quality for genre classification, emotion recognition, automatic tagging, and beat tracking, albeit with varying strength across tasks and pre-training paradigms. However, all metrics fail on tonal tasks such as key estimation and chord recognition, indicating that no single property serves as a general proxy for representation quality across music information retrieval tasks. To address this gap, we introduce a pitch-transposition equivariance measure that captures properties missed by these standard metrics, providing a consistent indicator of tonal quality across model families. Finally, we show that intrinsic metrics can serve as effective proxies for layer selection, matching or outperforming trainable multi-layer fusion methods, particularly in limited-data settings.
1. INTRODUCTION
This work systematically analyzes intrinsic layer-wise properties across 12 music foundation models and three pre-training paradigms. It evaluates whether these properties predict downstream layer quality and enable label-free layer selection across diverse MIR tasks.
- Systematic layer-wise analysis: The study analyzes intrinsic representation properties across 12 models spanning masked, autoregressive, and contrastive pre-training, revealing distinct depth-wise profiles.It focuses on how representation properties evolve across layers rather than relying only on downstream probes.
- Intrinsic metrics: Intrinsic dimension, curvature, and invariance-based metrics track downstream layer performance for several tasks but fail on tonal tasks.Their usefulness depends on both the downstream task and the pre-training paradigm.
- Tonal representation quality: Pitch-transposition equivariance is introduced to measure tonal representation quality missed by standard intrinsic metrics.The measure is motivated specifically by the failure of existing metrics on tonality.
- Practical layer selection: Intrinsic metrics support label-free layer selection and fusion, narrowing exhaustive probing to a few candidates while recovering near-oracle performance.They match or outperform trainable multi-layer fusion, particularly when labeled data is scarce.
2. METHODOLOGY
The study analyzes 12 frozen music foundation models across three pre-training paradigms using layer-wise representations extracted from 10,000 music clips. It evaluates layers through geometric, augmentation-conditioned, and pitch-transposition metrics across diverse downstream music tasks.
- Models: The analysis covers 12 publicly available music foundation models spanning masked, autoregressive, and contrastive pre-training paradigms.The masked family includes MERT variants, MusicFM, MuQ, MuQiter, and OMAR-RQ, with differing masked prediction targets.
- Representation extraction: Layer-wise representations are extracted from frozen models using 10,000 randomly sampled 15-second MTG-Jamendo clips, with mean-pooled vectors or full token sequences.Sequence-level metrics use aggregated clip vectors, whereas frame-level metrics retain token sequences.
- Intrinsic metrics: Intrinsic metrics characterize representation geometry, spectral structure, augmentation-conditioned behavior, and pitch-transposition equivariance.The geometric and spectral measures include intrinsic dimension, anisotropy, and temporal curvature; augmentation metrics assess discriminability and invariance.
- Augmentation-conditioned metrics: Augmented views apply pitch shifting, time stretching, Gaussian noise, gain variation, time shifting, and low-pass filtering to assess representation responses to controlled perturbations.LiDAR discounts directions unstable under augmentation, while InfoNCE measures agreement between augmented positive pairs relative to negatives.
- Pitch-transposition equivariance: Pitch-transposition equivariance is scored as PTE(l) = 1 − 1/2 d(l) ∈ [0, 1], where higher values indicate decodable 12-TET transposition structure.The score uses RMS chordal distance between normalized cross-power and its target after optimization.
3. EMPIRICAL FINDINGS
Layer quality varies by depth, task, and pre-training paradigm: intermediate layers often outperform final layers, while different intrinsic properties predict different task families. Proxy-guided layer selection and fusion can closely approach or exceed oracle and trainable baselines, despite weak cross-model ranking reliability.
- Layer-wise patterns: Intermediate layers consistently outperform final layers in masked and autoregressive models, with task-family performance peaks ordered by depth.The supplied passage specifies that tonal tasks peak earliest, at approximately 30% depth.
- Layer-wise patterns: Curvature is negatively correlated with downstream performance across rhythm, semantic, timbre, and similarity tasks, especially beat and downbeat tracking.Mean correlations are −0.69/−0.76 for masked models and −0.68/−0.67 for autoregressive models on beat/downbeat tracking.
- Tonal representations: PTE is the only consistently positive proxy for tonal-task performance, with strongest overall alignment at ¯ρ=0.80.Per-task correlations are ¯ρ=0.87 for pitch, ¯ρ=0.83 for key, and ¯ρ=0.59/0.63 for chord roots/triads.
- Cross-model limitations: Intrinsic metrics identify strong layers within models but do not reliably rank models across architectures, tasks, or pre-training paradigms.Effective rank and intrinsic dimension correlate at ρc=0.70 within masked encoders but reverse sign for autoregressive models at ρc=−0.47.
- Proxy-based layer selection: Averaging the top-3 proxy-ranked layers achieves a 0.3 pp mean gap to the oracle and improves over it in 41% of model–task pairs.The result is statistically stronger than trainable baselines, with p < 0.001 using an FDR-corrected Wilcoxon test.
- Proxy-based layer selection: For autoregressive backbones, proxy-guided averaging stays within 0.1 pp of the oracle, while every trainable fusion baseline has a gap of at least 2.8 pp.On the similarity task, top-k averaging also outperforms all-layer averaging by 4.8 pp.
4. CONCLUSIONS
Across three pre-training paradigms, no single intrinsic property universally predicts layer quality across music information retrieval tasks. The analysis is correlational, and existing models confound objective, supervision, data, architecture, and scale.
- Conclusions: No single intrinsic property provides a universal proxy for layer quality across MIR tasks.The analysis spans masked, autoregressive, and contrastive pre-training paradigms.
- Conclusions: Intrinsic dimension, curvature, and augmentation-conditioned metrics track downstream transfer with task- and paradigm-dependent strength but remain weak for tonal tasks.
- Limitations and future work: The correlational analysis does not establish whether directly manipulating these representation properties would improve downstream performance.Future work should test explicit regularization during pre-training for causal effects on representation quality and transfer.
- Limitations and future work: Objective, supervision, data, architecture, and scale co-vary across existing models, preventing their contributions from being disentangled.