Source-linked AI summary
Automatic tagging using deep convolutional neural networks
Keunwoo Choi, George Fazekas, Mark Sandler
TL;DR
Automatic music tagging requires representations whose relevance to tag categories is not known in advance. The paper evaluates fully convolutional networks with 2D convolutions and subsampling, finding competitive 4-layer mel-spectrogram performance on MagnaTagATune and better deeper-model performance on the larger Million Song Dataset.
Problem
Automatic tagging depends on acoustic features associated with target tags, but clear recommendations for relevant features have not emerged.
Method
The paper evaluates deep fully convolutional networks using 2D convolutional and subsampling layers with different depths and audio input representations.
Results
A 4-layer mel-spectrogram model performs competitively on MagnaTagATune, while deeper models outperform the 4-layer architecture on the larger Million Song Dataset.
Takeaways & Limitations
Mel-spectrograms are effective inputs for automatic tagging, and more complex models benefit from more training data.
Takeaways & Limitations
Very deep architectures are limited by computational resources and may require designs that support efficient training.
Abstract
from arXiv · showhide
We present a content-based automatic music tagging algorithm using fully convolutional neural networks (FCNs). We evaluate different architectures consisting of 2D convolutional layers and subsampling layers only. In the experiments, we measure the AUC-ROC scores of the architectures with different complexities and input types using the MagnaTagATune dataset, where a 4-layer architecture shows state-of-the-art performance with mel-spectrogram input. Furthermore, we evaluated the performances of the architectures with varying the number of layers on a larger dataset (Million Song Dataset), and found that deeper models outperformed the 4-layer architecture. The experiments show that mel-spectrogram is an effective time-frequency representation for automatic tagging and that more complex models benefit from more training data.
1. INTRODUCTION
Automatic tagging predicts descriptive music tags from audio, but conventional pipelines depend on hand-crafted features whose relevance is difficult to determine. The paper proposes fully convolutional networks to learn relevant representations automatically.
- Automatic tagging predicts music tags from audio through feature extraction followed by classification or regression.
- Conventional systems compute hand-crafted time- or frequency-domain features before machine learning, making task-relevant feature selection difficult.
- Feature learning unifies feature extraction and machine learning so deep neural networks can learn relevant features automatically.
- Deep convolutional networks learn hierarchical features and can provide local invariance and robustness to translation and distortion.
- The paper proposes an automatic music-tagging algorithm based on deep fully convolutional networks.
- The proposed architectures use convolutional and subsampling layers, reducing feature maps to 1×1 while modeling temporal and harmonic audio structure.
2. CNNS FOR MUSIC SIGNAL ANALYSIS
CNNs are applied to music through time-frequency representations and architectural choices that determine learned invariances. The section motivates mel-spectrograms, 2D convolutions, kernel sizes, and pooling as task-dependent design decisions.
- CNNs can learn robust hierarchical features from local time-frequency regions, but audio architecture must account for which invariances are desirable.
- Music tagging targets high-level song information, making hierarchical feature learning a natural fit for CNNs.
- Mel-spectrograms are widely used in tagging and other audio tasks, supported by auditory-system knowledge and empirical performance gains.
- Larger kernels can capture larger components, but increase parameters and reduce invariance within their receptive range.
- 2D convolutions learn temporal and spectral structures together, unlike time-axis convolution that learns temporal distributions separately by frequency band.
- Pooling reduces feature-map size while preserving regional activations, and its size trades location tolerance against the risk of missing features.
3. PROBLEM DEFINITION
Automatic tagging is a multi-label classification problem in which clips may receive multiple tags. Because crowd-sourced labels are highly imbalanced, the paper evaluates performance with AUC.
- Automatic tagging is multi-label classification because one music clip can receive multiple tags.
- With K exclusive labels, single-label classification has K possible outputs, whereas multiple labels can produce up to 2^K cases.
- Crowd-sourced tagging datasets contain mostly false labels for most clips, making accuracy and mean square error inappropriate measures.
- AUC is used because it is robust to unbalanced datasets and summarizes performance in one value.
4. PROPOSED ARCHITECTURE
The proposed FCNs use 2D convolutions and subsampling without fully connected layers to classify whole audio clips from mel-spectrograms. Variants deepen the architecture with additional layers, while pooling and parameter sharing support compact, location-invariant representations.
- FCN-4 has four convolutional and four max-pooling layers, takes a 96×1366 log-amplitude mel-spectrogram, and predicts a 50-dimensional tag vector.
- FCN-5, FCN-6, and FCN-7 extend the architecture, with FCN-6 and FCN-7 adding 1×1 convolutional layers for increased final-layer nonlinearity.
- The FCN-4 block diagram tracks feature-map channels, with subsampling shrinking maps to 1×1 and convolution increasing depth to 2048.
- Compared with related systems, the proposal uses 2D convolution and pooling to capture local harmonic structure and processes the whole 29.1s signal for temporal nonlinear aggregation.
- The architectures use only convolutional and subsampling layers, avoiding fully connected layers that account for most parameters and can promote overfitting.
5. EXPERIMENTS AND DISCUSSION
Experiments compare FCN architectures and audio representations on MagnaTagATune and the Million Song Dataset. Mel-spectrograms perform best among tested inputs, while deeper models benefit from the larger dataset.
- Experimental setup: The experiments evaluate FCN architectures with different layer counts and input representations on MagnaTagATune and the Million Song Dataset.Experiment I compares FCN-3, FCN-4, and FCN-5 with mel-spectrograms, plus STFT, MFCC, and mel-spectrogram inputs for FCN-4.
- Experimental setup: The MagnaTagATune evaluation uses 25,856 29.1-second clips and the 50 most frequent tags, with substantially imbalanced tag frequencies.The most frequent tag appears 4,851 times in training, compared with 490 occurrences for the 50th-ranked tag.
- Experiment I: MagnaTagATune: FCN-4 and FCN-5 show little performance difference on MagnaTagATune, while both perform competitively against previously reported systems.The authors attribute the limited gain from an additional layer most likely to the layer being unnecessary for the task rather than to gradient-flow problems.
- Experiment I: MagnaTagATune: Mel-spectrogram inputs substantially outperform STFT and MFCC inputs, and high-resolution low-frequency information helps automatic tagging in these tasks.The authors link this result to the allocation of 96 frequency bins across 1 kHz bands and suggest that low-frequency-focused downsampling can be more efficient.
- Experiment II: Million Song Dataset: The Million Song Dataset contains more than nine times as many songs as MagnaTagATune, enabling evaluation of deeper FCN architectures with last.fm tags.The experiment uses 214,284 selected training and validation clips plus 25,940 test clips after filtering for the top 50 tags.
- Experiment II: Million Song Dataset: On the Million Song Dataset, FCN-5, FCN-6, and FCN-7 significantly outperform FCN-3 and FCN-4, indicating that more complex models benefit from sufficient training data.FCN-6 exceeds FCN-5 by AUC 0.003, while FCN-7 performs slightly worse than FCN-6; computational resources constrain very deep structures.
6. CONCLUSION
The paper presents deep fully convolutional networks for automatic music tagging, finding that mel-spectrograms perform best and deeper networks benefit from larger training datasets.
- Deep fully convolutional networks with 2D convolutions can be used effectively for automatic music tagging and classification.
- Mel-spectrograms performed better than STFTs and MFCCs as audio input representations.
- The Million Song Dataset contains nine times as many music clips as the MagnaTagATune dataset.
- The optimal number of layers differed on the Million Song Dataset, indicating that deeper networks benefit from abundant training data.
- Future work will investigate automatic tagging algorithms with variable input lengths.