Source-linked AI summary
Convolutional Recurrent Neural Networks for Music Classification
Keunwoo Choi, George Fazekas, Mark Sandler, Kyunghyun Cho
TL;DR
Music tagging requires models that can represent both local audio features and temporal structure, motivating a comparison between CRNNs and existing CNNs. The paper combines convolutional extraction with recurrent summarisation and evaluates matched network sizes and computation settings. CRNN performs comparably to k2c2 with a modest number of parameters, while speed and memory trade-offs vary across parameter settings.
Problem
Music tags may depend on either global temporal structure or local short-segment information, motivating models that can handle both.
Method
The paper compares a CRNN with k1c2, k2c1, and k2c2 using controlled hardware, data, optimisation, parameter counts, and computation time.
Results
CRNN and k2c2 perform comparably with a modest number of parameters; k2c2 is faster across parameter settings, while CRNN tends to outperform it at equal parameter counts.
Takeaways & Limitations
CRNN is effective when memory is the bottleneck, while k2c2 or CRNN can be selected according to the target time budget.
Takeaways & Limitations
The CRNN design assumes temporal patterns can be aggregated better with RNNs than CNNs and uses weak dropout to prevent overfitting of RNN layers.
Abstract
from arXiv · showhide
We introduce a convolutional recurrent neural network (CRNN) for music tagging. CRNNs take advantage of convolutional neural networks (CNNs) for local feature extraction and recurrent neural networks for temporal summarisation of the extracted features. We compare CRNN with three CNN structures that have been used for music tagging while controlling the number of parameters with respect to their performance and training time per sample. Overall, we found that CRNNs show a strong performance with respect to the number of parameter and training time, indicating the effectiveness of its hybrid structure in music feature extraction and feature summarisation.
1. INTRODUCTION
The paper motivates CRNNs for music tagging by combining CNN-based local feature extraction with RNN-based temporal summarisation. This hybrid structure can accommodate tags influenced by either global structure or short local segments, and is evaluated against existing CNNs under controlled conditions.
- Background: CNNs have been applied to music tagging, genre classification, and user-item latent feature prediction for recommendation.CNNs learn hierarchical features ranging from low-level onsets to high-level percussive instrument patterns.
- CRNN motivation: CRNNs replace later CNN layers with an RNN, using CNNs for local feature extraction and RNNs for temporal summarisation.The RNN aggregation incorporates global structure while convolutional layers extract local features.
- CRNN motivation: RNN-based summarisation is more flexible than CNN-based weighted averaging and subsampling for music tags with different temporal characteristics.Mood tags may depend on global structure, whereas instrument tags may depend on local, short-segment information.
- Contribution: The paper introduces CRNNs for music tagging and compares them with three existing CNNs while controlling hardware, data, optimisation, parameter count, and computation time.The structural attributes varied are the number of parameters and computation time.
2. MODELS
The compared models use different convolutional structures, while CRNN adds recurrent temporal summarisation to a two-dimensional CNN. The study scales networks by parameter count while preserving depth and kernel shapes to compare structure and capacity.
- Model comparison: The comparison includes k1c2, k2c1, k2c2, and CRNN, using 96×1366 mel-frequency-band-by-time-frame single-channel inputs and sigmoid outputs for multi-label tagging.The CNN names encode kernel shape and convolution dimension.
- Training setup: All models use batch normalization and ELU activations, while CRNN additionally uses weak dropout of 0.1 between convolutional layers.The shared optimisation and activation settings support controlled comparisons; the dropout is used to prevent overfitting of RNN layers.
- CNN structures: k2c1 compresses the whole frequency range into one band in its first convolutional layer, greatly reducing computation complexity.Its later convolutions operate along the time axis.
- CNN structures: k2c2 uses five 3×3 convolutional layers with gradual two-dimensional subsampling, producing final features that cover the whole input.The fully-convolutional design supports multi-scale time-frequency invariance and fewer parameters.
- CRNN: CRNN uses a 2-layer GRU to summarise temporal patterns on top of a 4-layer two-dimensional CNN.Its convolutional and pooling stages produce feature maps of size N×1×15 before the recurrent layers.
- Scaling networks: The networks are scaled to 100,000, 250,000, 0.5 million, 1M, and 3M parameters with 2% tolerance.Layer widths change while depths and convolutional kernel shapes remain constant, preserving the hierarchy of learned features.
3. EXPERIMENTS
Experiments evaluate CRNN and three CNN structures on top-50 music tags while controlling parameter count, training time, hardware, data, and optimisation. CRNN leads at matched parameter counts, while k2c2 and CRNN provide the strongest performance at similar training times.
- Experimental setup: The study predicts 50 genre, mood, instrument, and era tags from Million Song Dataset preview clips using AUC-ROC.The dataset contains 214,284 selected training/validation clips and 25,940 test clips; inputs are 96×1366 log-amplitude mel-spectrograms.
- Memory-controlled experiment: At equal parameter counts, AUC ranks CRNN > k2c2 > k1c2 > k2c1, making CRNN preferable when memory is the bottleneck.CRNN outperforms k2c2 in all tested parameter-count cases, including 0.1M versus 0.25M parameters.
- Memory-controlled experiment: k2c2 exceeds k2c1 and k1c2 in AUC across all parameter counts, while its fully convolutional structure allocates parameters more flexibly.For k1c2 with 0.5M parameters, 13% support convolutional layers and 87% support fully connected layers.
- Computation-controlled comparison: At similar training times, k2c2 and CRNN show the best performance, whereas training speed at equal parameter counts ranks k2c1 > k2c2 > k1c2 > CRNN.The speed ranking is attributed to feature-map sizes, convolutional computation, and CRNN depth reaching up to 20.
- Performance per tag: CRNN outperforms k2c1 for 44 tags, while k2c1 outperforms k1c2 for 48 of 50 tags.Tag popularity is not correlated with AUC ranking; the Spearman rank correlation is 0.077.
4. CONCLUSIONS
CRNN and k2c2 perform comparably with a modest number of parameters, while their relative speed and memory use trade off at very small or large sizes. k2c2 is faster across settings, but CRNN tends to outperform it when parameter counts are matched.
- CRNN and k2c2 perform comparably with a modest number of parameters.
- At very small or large parameter counts, CRNN and k2c2 exhibit a trade-off between speed and memory.
- k2c2 computes faster across all parameter settings, while CRNN tends to outperform it with the same number of parameters.