Source-linked AI summary
End-to-end learning for music audio tagging at scale
Jordi Pons, Oriol Nieto, Matthew Prockup, Erik Schmidt, Andreas Ehmann, Xavier Serra
TL;DR
Deep music auto-tagging is limited by the availability of large annotated datasets, especially for end-to-end waveform models. The paper compares assumption-free waveform and domain-informed spectrogram architectures across datasets of different sizes. Its central result is that domain knowledge is relevant with limited data, whereas waveform models outperform spectrogram models when enough data are available.
Problem
The study addresses how deep music auto-tagging architectures with different design assumptions perform as the amount of annotated training data varies.
Method
The paper compares a raw-waveform sample-level model with a log-mel spectrogram model guided by musical domain knowledge across three differently sized datasets.
Results
Given enough data, assumption-free waveform models outperform domain-informed spectrogram models, while domain knowledge remains relevant when sizable datasets are unavailable.
Takeaways & Limitations
Architectural assumptions are useful in data-scarce music auto-tagging settings, but large datasets can support effective waveform-based models without musical domain assumptions.
Takeaways & Limitations
The models leave data imbalance and alternative temporal aggregation strategies for future improvement.
Abstract
from arXiv · showhide
The lack of data tends to limit the outcomes of deep learning research, particularly when dealing with end-to-end learning stacks processing raw data such as waveforms. In this study, 1.2M tracks annotated with musical labels are available to train our end-to-end models. This large amount of data allows us to unrestrictedly explore two different design paradigms for music auto-tagging: assumption-free models - using waveforms as input with very small convolutional filters; and models that rely on domain knowledge - log-mel spectrograms with a convolutional neural network designed to learn timbral and temporal features. Our work focuses on studying how these two types of deep architectures perform when datasets of variable size are available for training: the MagnaTagATune (25k songs), the Million Song Dataset (240k songs), and a private dataset of 1.2M songs. Our experiments suggest that music domain assumptions are relevant when not enough training data are available, thus showing how waveform-based models outperform spectrogram-based ones in large-scale data scenarios.
1. INTRODUCTION
Music auto-tagging aims to organize large collections by estimating semantic musical attributes. This work compares assumption-free and domain-informed deep architectures across datasets of varying size.
- Motivation: Music auto-tagging estimates attributes such as mood, language, genre, instruments, harmony, and rhythm.These estimates can define a semantic space for automatically organizing musical libraries.
- Research question: The study compares two deep architectures based on opposite design strategies across several datasets, including an unprecedented 1.2M-song collection.One strategy uses domain knowledge, while the other minimizes task-specific assumptions.
- Research question: The central question is whether domain-informed choices such as log-mel spectrogram inputs constrain learning when abundant training data are available.The authors frame the contribution as showing that little to no model assumptions are required for large-scale music auto-tagging.
2. CURRENT DEEP ARCHITECTURES
The paper separates deep audio models into front-ends that map inputs into latent representations and back-ends that predict outputs from them. Front-ends differ by input type and by whether their filters incorporate domain knowledge.
- Model decomposition: A front-end interacts with the input signal to produce a latent-space representation, while a back-end predicts outputs from that representation.This division provides a framework for discussing audio architectures.
- Front-ends: Front-ends are generally CNNs and use either waveforms or spectrograms as input.Their filters may be designed with or without domain knowledge.
- Front-ends: Domain-informed filter designs connect deep learning architectures with signal-processing concepts such as STFT windows, multiresolution transforms, and time-frequency scales.Without domain knowledge, the architecture makes fewer such explicit assumptions.
- Back-ends: Back-ends are grouped into fixed-length and variable-length input designs.Convolutional front-ends can process varying input lengths, after which the back-end adapts feature maps to fixed-sized outputs.
3. DATASETS
The study evaluates music auto-tagging architectures on three collections spanning roughly 26k, 240k, and 1.2M songs. The datasets include public benchmarks and a privately assembled large-scale collection.
- Dataset overview: The experiments compare architectures across three music collections with different dataset sizes.This design supports analysis of how training-set scale relates to model behavior.
- Million Song Dataset: MSD provides approximately 240k 30-second previews with proper top-50 tag annotations, making it the largest public music auto-tagging benchmark here.The dataset name refers to 1M songs, but only the annotated previews are used.
- Private dataset: The private collection contains 1M training songs, 100k validation songs, and 100k test songs with track-level human-expert annotations.Its total scale is 1.2M songs, and the partitions are artist-filtered and stratified.
- Annotations: The annotation vocabulary covers instrumentation, sonority, genres and subgenres, meter, rhythmic feel, harmony, mood, and vocals.These categories describe musical content, production, and perceptual characteristics.
4. THE ARCHITECTURES UNDER STUDY
The study selects two contrasting front-ends while holding their back-end constant for fair comparison: a sample-level waveform model and a domain-informed spectrogram model. Both process 15-second audio segments, with the spectrogram branch targeting timbral and temporal patterns.
- Waveform front-end: The waveform model feeds raw audio directly into a sample-level front-end using very small 3×1 convolutional filters.Its design makes minimal assumptions about which waveform patterns are informative.
- Design paradigms: The selected models represent opposite paradigms: assumption-free waveform processing versus spectrogram processing guided by musical domain knowledge.The shared back-end enables a fair comparison of the selected front-ends.
- Spectrogram front-end: The spectrogram model converts 15-second audio into normalized log-mel spectrograms with 96 mel bins.Its first-layer vertical and horizontal filters are designed around timbral and temporal structure.
- Spectrogram front-end: The spectrogram front-end uses separate branches for timbral and temporal features.The timbral branch captures pitch-invariant patterns across time-frequency scales, while the temporal branch captures multiple time scales.
- Training and prediction: Training uses 15-second patches, while song-level prediction averages outputs from multiple moving windows.The basic configuration also specifies dropout, ReLU nonlinearities, Adam optimization, and dataset-dependent losses.
5. EXPERIMENTAL RESULTS
Across datasets of different sizes, spectrogram models are stronger with scarce data, while waveform models become superior at large scale. The experiments also examine model capacity, qualitative predictions, and remaining optimization limitations.
- 1.2M-songs dataset: 1M training tracks produce deep-learning models that outperform the baseline in every metric, whereas 100k-track models perform worse.The largest improvement relative to the baseline occurs for PR-AUC, which is more informative for unbalanced data.
- 1.2M-songs dataset: At 1M songs, the waveform model outperforms the spectrogram model in every metric.The authors relate this result to the waveform front-end's less constrained solution space.
- 1.2M-songs dataset: With 100k songs, spectrogram models tend to perform better; with 1M songs, waveform models outperform them across ROC-AUC, PR-AUC, and MSE.Linear trend fits in Figure 3 are used to examine this change with training-set size.
- 1.2M-songs dataset: Qualitatively, the deep model is biased toward popular tags but better differentiates similar mutually exclusive tags than the independent-GBT baseline.The popular-tag bias is expected because training does not address data imbalance.
- MagnaTagATune dataset: On MagnaTagATune, both model families outperform the proposed state-of-the-art baselines, but spectrogram models perform better than waveform models.Performance is relatively robust to parameter count, with only minor differences between small and large models.
- Million Song Dataset: On the Million Song Dataset, the spectrogram model outperforms the waveform model and performs equivalently to the best literature method, Multi-level & multiscale.The authors suggest differences in the number of song-level estimates or global pooling may contribute to the waveform model's lower performance.
6. CONCLUSIONS
The study finds that domain knowledge is useful when datasets are small, but with enough data, assumption-free waveform models outperform spectrogram-based models. The MSD results table separates waveform-based models from spectrogram-based models for comparison.
- Given enough data, assumption-free waveform models outperform models relying on musical domain knowledge.
- Domain-knowledge-based models remain relevant when sizable datasets are unavailable.
- The two competitive architectures use contrasting principles: waveform inputs without domain-knowledge design versus spectrogram inputs justified by musical knowledge.
- Table 4 places waveform-based models at the top and spectrogram-based models at the bottom of the MSD results.