Source-linked AI summary
Sample-level Deep Convolutional Neural Networks for Music Auto-tagging Using Raw Waveforms
Jongpil Lee, Jiyoung Park, Keunhyoung Luke Kim, Juhan Nam
TL;DR
Music auto-tagging commonly depends on engineered spectrogram features, and prior raw-waveform CNNs had not fully matched them. The paper introduces deeper sample-level DCNNs that learn from waveform filters as small as 2 or 3 samples, achieving comparable results to prior state of the art on two datasets while visualizing hierarchical filters.
Problem
Prior raw-waveform CNNs for music auto-tagging struggled with model expressiveness, amplitude compression, and frame-level bottom-layer processing.
Method
The paper uses sample-level DCNNs with very small first-layer filters and strides, increasing network depth to learn representations directly from raw waveforms.
Results
The sample-level architecture achieves results comparable to previous state-of-the-art performances on the MagnaTagATune and Million Song Dataset.
Takeaways & Limitations
Deeper models with very small sample-level filters and subsampling lengths are more effective for music auto-tagging, while learned filters can be visualized hierarchically.
Abstract
from arXiv · showhide
Recently, the end-to-end approach that learns hierarchical representations from raw data using deep convolutional neural networks has been successfully explored in the image, text and speech domains. This approach was applied to musical signals as well but has been not fully explored yet. To this end, we propose sample-level deep convolutional neural networks which learn representations from very small grains of waveforms (e.g. 2 or 3 samples) beyond typical frame-level input representations. Our experiments show how deep architectures with sample-level filters improve the accuracy in music auto-tagging and they provide results comparable to previous state-of-the-art performances for the Magnatagatune dataset and Million Song Dataset. In addition, we visualize filters learned in a sample-level DCNN in each layer to identify hierarchically learned features and show that they are sensitive to log-scaled frequency along layer, such as mel-frequency spectrogram that is widely used in music classification systems.
1. INTRODUCTION
Music information retrieval commonly relies on engineered time-frequency representations, while end-to-end learning from raw waveforms remains less fully explored. The paper addresses limitations of prior raw-waveform music auto-tagging with sample-level DCNNs.
- MIR systems typically convert music waveforms into engineered log-scaled time-frequency representations such as mel-spectrograms or constant-Q transforms.
- Deep convolutional networks can learn hierarchical representations directly from raw input, reducing manually designed input processing.
- Prior raw-waveform CNNs did not match mel-spectrogram systems, partly because their models, nonlinearities, and frame-level bottom layers were inadequate.
- Sample-level DCNNs use bottom-layer filters as small as several samples, with reduced first-layer strides and increased network depth.
- The proposed architectures achieved results comparable to previous state-of-the-art performances on the MagnaTagATune and Million Song Dataset benchmarks.
2. RELATED WORK
Earlier waveform-based CNNs used frame-level strided convolutions, whereas this work reduces the first-layer filter length and stride to sample-level granularity and compensates with greater depth.
- Prior waveform-input CNNs used a large strided convolution in the bottom layer, analogous to hopping windows over the audio waveform.
- The proposed approach reduces the first convolution’s filter length and stride to as little as 2 samples, then increases CNN depth.
- Earlier models using 10-sample strides had only three convolutional layers, which the authors describe as insufficient for complex musical structure.
3. LEARNING MODELS
The paper compares mel-spectrogram, frame-level waveform, and sample-level waveform CNNs, designing the latter with small filters, repeated subsampling modules, and increased depth.
- The experiments compare frame-level mel-spectrogram, frame-level raw-waveform, and sample-level raw-waveform CNN models.
- Frame-level mel-spectrogram model: The frame-level mel-spectrogram model uses a one-dimensional CNN because its performance is not significantly different from a two-dimensional alternative.
- Frame-level raw waveform model: The frame-level waveform model adds strided convolution intended to learn a filter-bank representation corresponding to the time-frequency input dimensions.
- Sample-level raw waveform model: The sample-level design adds multiple small convolution and pooling layers so the first layer can process much shorter waveform segments.
- Model Design: Input size, hidden-layer temporal length, and convolutional filter and subsampling lengths are treated as key architectural configuration issues.
- Model Design: The mn-DCNN notation uses m for intermediate filter and pooling length and n for the number of modules, or network depth.
4. EXPERIMENTAL SETUP
The experiments evaluate sample-level waveform models against established approaches on two music-tagging datasets, using controlled configurations and standard training settings.
- Experimental configuration: Table 1 specifies sample-level CNN layers through filter length, filter count, and pooling length.
- Datasets: The study evaluates the proposed model on MagnaTagATune and the larger Million Song Dataset, using the 50 most frequently labeled tags in each.
- Datasets: All songs in both datasets were trimmed to 29.1 seconds and resampled for the experiments.
- Experimental configuration: Table 2 varies input size, intermediate filter and pooling length m, module count n, and the first-layer filter length and stride.
- Training settings: The models use sigmoid outputs with binary cross entropy, batch normalization and ReLU in convolutional layers, dropout of 0.5, and stochastic gradient descent.
- Training settings: Mel-spectrogram inputs were normalized, whereas raw-waveform inputs were not.
5. RESULTS
The experiments evaluate sample-level raw-waveform DCNNs across model configurations, mel-spectrogram comparisons, state-of-the-art baselines, and learned-filter visualizations. Deeper architectures with small filters perform strongly, while learned filters show frequency-structured behavior across layers.
- mn-DCNN models: Input lengths of 1–3 seconds are reasonable choices for raw-waveform networks, similarly to mel-spectrogram models.
- Mel-spectrogram and raw waveforms: The sample-level raw-waveform model achieves results comparable to the frame-level mel-spectrogram model.A hop size of 81 samples, approximately 4 ms, performs better than conventional mel-spectrogram hop sizes of about 20 ms, whereas shorter hops degrade performance.
- Comparison to state-of-the-arts: The proposed sample-level architecture is evaluated against prior state-of-the-art systems on MTAT and MSD and is reported as highly effective compared with them.
- Visualization of learned filters: Gradient ascent visualization produces waveforms that maximize target-filter activations, enabling inspection of learned filters across layers.The estimated filters are converted to spectra and sorted by peak frequency because waveform estimates remain noisy.
- Visualization of learned filters: Higher layers show increasingly nonlinear spacing of filter-bank center frequencies, with steeper spacing in the high-frequency region.The pattern resembles perceptual pitch scales such as mel or bark.
6. CONCLUSION AND FUTURE WORK
The paper concludes that deep sample-level DCNNs can learn effective representations directly from raw waveforms for music auto-tagging. It also visualizes hierarchical filters and identifies further analysis of the architecture and learned filters as future work.
- 6. CONCLUSION AND FUTURE WORK: Deeper models with more than 10 layers and very small sample-level filters and subsampling lengths are more effective for music auto-tagging.The reported results are comparable to previous state-of-the-art performances on MTAT and MSD.
- 6. CONCLUSION AND FUTURE WORK: The paper visualizes hierarchically learned filters to examine representations learned from raw waveforms.
- 6. CONCLUSION AND FUTURE WORK: Future work will investigate why the architecture works without input normalization or amplitude-compressing nonlinear functions and will analyze the learned filters further.