Source-linked AI summary
Transfer learning for music classification and regression tasks
Keunwoo Choi, György Fazekas, Mark Sandler, Kyunghyun Cho
TL;DR
The paper addresses how to obtain transferable music representations for classification and regression when target tasks differ. It trains a music-tagging convnet, concatenates pooled activations across layers, and transfers the resulting features. Across six tasks, the convnet feature outperforms MFCCs and remains competitive with task-specific state-of-the-art methods.
Problem
The paper seeks a general-purpose music representation that can transfer across music classification and regression tasks.
Method
A convnet is pretrained for music tagging, then pooled activations from multiple layers are concatenated and transferred as features to target tasks.
Results
Across all six tasks, the pretrained convnet feature outperformed MFCCs and was competitive with state-of-the-art methods designed for individual tasks.
Takeaways & Limitations
Aggregating features across convolutional layers generally provides an effective transferable representation, although the most important layer differs by task.
Takeaways & Limitations
For acoustic event classification, the music-trained features did not outperform a dedicated target-task convnet, whose unaugmented accuracy was reported as 74% versus 71.4% for the best proposed result.
Abstract
from arXiv · showhide
In this paper, we present a transfer learning approach for music classification and regression tasks. We propose to use a pre-trained convnet feature, a concatenated feature vector using the activations of feature maps of multiple layers in a trained convolutional network. We show how this convnet feature can serve as general-purpose music representation. In the experiments, a convnet is trained for music tagging and then transferred to other music-related classification and regression tasks. The convnet feature outperforms the baseline MFCC feature in all the considered tasks and several previous approaches that are aggregating MFCCs as well as low- and high-level music features.
1. INTRODUCTION
Transfer learning reuses parameters trained on a source task to transfer knowledge to a target domain, especially when target-task data are limited. Prior examples span computer vision, natural language processing, and music information retrieval.
- Transfer learning reuses source-task parameters for a target task to transfer knowledge between domains.
- Transferring pre-trained neural-network weights can reduce the number of trainable target-task parameters and support learning with smaller datasets.
- Computer-vision segmentation and pretrained word embeddings illustrate transfer across related downstream tasks.
- Earlier MIR studies transferred learned representations or weights from music tagging, play-count, and related source data to genre classification, auto-tagging, recommendation, and other tasks.
2. TRANSFER LEARNING FOR MUSIC
The paper trains a convolutional network for music tagging and transfers its representations to target tasks. Instead of relying only on the final layer, it aggregates features from intermediate and final layers to preserve complementary information.
- Approach: A convnet is trained on music tagging and then used as a feature extractor for target classification and regression tasks.
- Source task: Music tagging is selected as the source task because large training data and labels covering genres, moods, eras, and instrumentations are available.
- Network: The network uses five convolutional and subsampling layers with two-dimensional 3×3 kernels to learn hierarchical time-frequency patterns.
- Representation transfer: The proposed representation combines activations from up to all intermediate layers rather than using only the final hidden layer, motivated by potentially discarded low-level information.
- Feature extraction: During transfer, average pooling reduces layers 1–4 to 1×1 summaries, while the fifth-layer feature is already scalar; these 32-dimensional vectors are concatenated.
- Controls: The study evaluates random convnet features to separate contributions from pretrained knowledge transfer, network structure, and nonlinear transformation.
- Target models: SVM classifiers and regressors are used because target tasks have small training sets and to focus comparisons on feature performance.
3. PREPARATION
The evaluation pretrains the source convnet on Million Song Dataset clips and transfers its features to six music and audio tasks. MFCC statistics provide the baseline, while random weights provide a structural control.
- Source task: 244,224 Million Song Dataset preview clips train, validate, and test the music-tagging source task using 50 Last.fm tags.
- Source task: The source convnet receives log-scaled 96-bin mel-spectrograms and is trained with binary cross-entropy using ADAM optimization.
- Target tasks: Six target datasets cover ballroom genre, Gtzan genre, a smaller task dataset, emotion prediction, vocal/non-vocal classification, and acoustic events.
- Target tasks: The vocal-detection data are used as a pre-segmented classification task, which may be easier than the original frame-based benchmarking task.
- Baselines: MFCC baseline vectors contain means and standard deviations of 20 coefficients plus first- and second-order derivatives.
- Baselines: Random convnet features use the identical architecture with normally initialized random weights and no training.
4.1 Configurations
The experiments compare convnet-feature configurations with MFCCs and reported state-of-the-art results across six tasks. Cross-validation, predefined splits, and validation-based SVM tuning govern the evaluation.
- Evaluation: For Tasks 1–4, experiments use 10-fold cross-validation with stratified splits.
- Evaluation: Task 5 uses predefined training, validation, and test sets, while Task 6 uses 10-fold cross-validation without replacement to avoid recording-level subsegment overlap.
- Model selection: SVM kernel type, radial-basis bandwidth, and penalty parameter are selected by grid search based on validation results.
- Model selection: The radial-basis function is exp(−γ|x−x′|2), with γ denoting bandwidth and Nf denoting feature-vector dimensionality.
4.2 Results and Discussion
Across six tasks, most convnet feature configurations outperform the baseline feature, while concatenating all layers with MFCCs usually adds no benefit. Performance varies by task: selected layers achieve strong genre and emotion results, random features can remain competitive, and dedicated target-task models may still perform better.
- Most convnet feature configurations outperform the baseline feature across all six tasks.
- Concatenating the ‘12345’ convnet feature with MFCCs usually does not improve over a pure convnet feature, except for audio event classification.
- Task 1. Ballroom Genre Classification: 86.7% accuracy is achieved by the ‘123’ convnet feature in Ballroom dance classification.The result also outperforms some previous approaches using explicit rhythmic features.
- Task 2. Gtzan Music Genre Classification: 89.8% accuracy is achieved by the convnet feature on Gtzan genre classification, versus 78.1% for concatenated features and 66.0% for MFCCs.The ‘12345’, ‘2345’, and ‘1234’ configurations are among the strongest, with all top-seven configurations using more than three layers.
- Task 3. Gtzan Speech/music Classification: 100% accuracy is achieved by a majority of convnet feature combinations for Gtzan speech/music classification, compared with 99.2% for MFCCs.The authors suggest the pre-trained feature easily solves this task because the music and speech signals are highly distinctive in the dataset.
- Task 4. Music Emotion Prediction: The best convnet features achieve r2 scores of 0.633 for arousal and 0.415 for valence in music emotion prediction.Arousal relies strongly on the fifth layer, whereas valence favors the third layer, so the two axes should be optimized separately.
5. CONCLUSIONS
The paper proposes transferring a music-tagging convnet to six music information retrieval and audio-related tasks using pooled features from convolutional layers. The pre-trained representation outperformed MFCC and random-weight baselines across all six tasks, while layer usefulness varied by task and concatenating all layers generally worked well.
- Conclusions: The approach transfers a convnet trained for music tagging to genre, vocal/non-vocal, emotion, speech/music, and acoustic-event tasks.Features from convolutional layers are average-pooled to reduce feature-map sizes before transfer.
- Conclusions: Pre-trained convnet features outperformed baseline MFCC and random-convnet features on all six evaluated tasks.The comparison isolates both the representation’s advantage over a popular handcrafted baseline and the contribution of source-task pre-training.
- Conclusions: Layer importance differed across tasks, but concatenating features from all layers generally produced strong performance.The paper reports that the most important layer varied by task, motivating multi-layer representations rather than relying only on the final layer.
- Conclusions: Adding MFCCs to convnet features did not improve performance on the five music tasks.This indicates that the information useful from MFCCs was already represented in the convnet feature for those tasks.