PaperlayerView exact revision on arXiv

Source-linked AI summary

Multi-Level and Multi-Scale Feature Aggregation Using Pre-trained Convolutional Neural Networks for Music Auto-tagging

Jongpil Lee, Juhan Nam

arXiv:1703.01793v2cs.NEcs.LGcs.MMcs.SD

TL;DR

Music auto-tagging must handle tags that differ in abstraction and temporal scope, while much prior work used a common feature level or scale. The paper trains CNNs on multiple input sizes, aggregates features across layers and long clips, and reports strong results across auto-tagging and transfer-learning evaluations.

  • Problem

    Music tags range from local sound-source descriptions to broad genre or mood concepts, but most prior work predicted them from a common feature level or scale.

    Source: §I · p. 1

  • Method

    The method trains supervised CNNs at different input sizes, aggregates layer features over long clips, and uses a global classifier to predict tags.

    Source: §I · p. 1, A. Local feature learning · p. 2

  • Results

    The architecture achieves the best results on three popularly used datasets, and global models consistently outperform local models on MTAT and MSD.

    Source: A. Comparison of local and global models · p. 3, §V · p. 4

  • Takeaways & Limitations

    The separated feature-learning, aggregation, and classification stages support transfer learning with CNNs pre-trained on a large dataset.

    Source: §I · p. 1

Abstract

from arXiv · show

Music auto-tagging is often handled in a similar manner to image classification by regarding the 2D audio spectrogram as image data. However, music auto-tagging is distinguished from image classification in that the tags are highly diverse and have different levels of abstractions. Considering this issue, we propose a convolutional neural networks (CNN)-based architecture that embraces multi-level and multi-scaled features. The architecture is trained in three steps. First, we conduct supervised feature learning to capture local audio features using a set of CNNs with different input sizes. Second, we extract audio features from each layer of the pre-trained convolutional networks separately and aggregate them altogether given a long audio clip. Finally, we put them into fully-connected networks and make final predictions of the tags. Our experiments show that using the combination of multi-level and multi-scale features is highly effective in music auto-tagging and the proposed method outperforms previous state-of-the-arts on the MagnaTagATune dataset and the Million Song Dataset. We further show that the proposed architecture is useful in transfer learning.

I. INTRODUCTION

Music tags span local, source-specific descriptions and broader concepts that require longer audio segments, while much prior work used a single feature level or scale. The paper proposes supervised multi-level, multi-scale CNN features aggregated for song-level tag prediction, and reports improvements over prior state-of-the-art systems.

I. INTRODUCTION

  • Instrument tags describe local, repetitive sound sources, whereas genre, mood, and era tags describe overall content and require longer audio segments.This diversity in tag abstraction and temporal scope motivates using features at multiple levels and scales.Source: §I · p. 1
  • Most prior work predicted tags from a common feature level or scale, while earlier attempts addressed hierarchical or time-scale features with narrower evaluations or input-only multiscaling.The cited work includes small-dataset evaluations, single-frame features, and multiscaling focused on spectrogram inputs.Source: §I · p. 1
  • The architecture trains CNNs on different input sizes, aggregates features from their layers across long clips, and uses a fully connected network for final tag predictions.The separated feature-learning and classification stages also support transfer learning across datasets.Source: §I · p. 1, Figure 1 · p. 2
  • Experiments report that combining multi-layer and multi-time-scale features improves accuracy and outperforms previous state-of-the-art systems on the evaluated auto-tagging datasets.The paper evaluates MagnaTagATune and Million Song Dataset.Source: §I · p. 1

II. PROPOSED METHOD

The proposed music auto-tagging architecture is presented as a sequence of three training steps.

II. PROPOSED METHOD

  • The proposed architecture is organized into three training steps, which the following method sections describe.The supplied passage introduces the sequence without naming its stages.Source: §II · p. 2

A. Local feature learning

The method learns local audio features with supervised CNNs and aggregates layer-specific features across input scales into song-level representations.

A. Local feature learning

  • Supervised CNNs use different input-frame sizes so their hidden layers capture audio features over different numbers of musical beats.The 18-, 27-, and 54-frame inputs span 420, 630, and 1260 msec, covering at least one beat across 48–143 BPM.Source: A. Local feature learning · p. 2
  • CNNs are adapted to input size, trained with dataset tag labels, and can serve as local models by averaging predictions across a long clip.Models above 54 input frames add convolutional layers; the 18-frame model uses a modified final convolutional layer.Source: A. Local feature learning · p. 2

B. Multi-level and multi-scale feature aggregation

  • For long clips, the method max-pools features within each segment, average-pools across the clip separately for each layer, then concatenates features across layers and input sizes.For example, the 27-frame model yields a 128 + 128 + 256 dimensional feature vector before combination with other input sizes.Source: B. Multi-level and multi-scale feature… · p. 2

C. Global classification

A global classifier predicts tags from aggregated features, with evaluation spanning auto-tagging datasets and transfer-learning genre-classification settings.

C. Global classification

  • A neural network with two fully connected hidden layers predicts tags from aggregated multi-level and multi-scale features, with 512 or 1024 units depending on the dataset.Separating aggregation and global classification from local feature learning permits transfer learning with pre-trained CNNs.Source: C. Global classification · p. 2

A. Datasets

  • Evaluation uses MagnaTagATune and Million Song Dataset for 50-tag auto-tagging, plus fault-filtered GTZAN and Tagtraum genre classification with MSD-pretrained CNN features.The auto-tagging metric is AUC; the genre tasks use 10 GTZAN genres and 15 Tagtraum genres.Source: A. Datasets · pp. 2–3
  • The figures compare local and global models across input sizes and combinations of feature levels in the 27-frame model.The captions specify comparison scope but do not provide the outcomes.Source: Figure 2 · p. 3, Figure 3 · p. 3

B. Training details

The supplied passages report the spectrogram representation and training settings, a comparison of multi-scale feature combinations, and the total MSD training time.

B. Training details

  • The models use 128-bin mel-frequency spectrograms, sampled at 22.05 kHz with a 512-sample hop and 1024-sample Hanning window.Magnitude compression uses log(1 + C|A|), with C set to 10.Source: B. Training details · p. 3
  • Fig. 4 compares various combinations of multi-scale features.Source: Figure 4 · p. 3
  • Training on MSD took about five days in total.Source: B. Training details · p. 3

C. Compared models

The section distinguishes local models, which average predictions from short audio segments, from global models, which aggregate local-model features to predict tags for a full clip.

C. Compared models

  • Local models take roughly three-second audio segments and average their outputs to predict tags for a clip.Source: C. Compared models · p. 3
  • Global models aggregate features from local models and make predictions directly from the audio clip.Source: C. Compared models · p. 3
  • The comparison evaluates both model types across input sizes and feature-level combinations.Source: C. Compared models · p. 3

IV. RESULTS AND DISCUSSION

Global models consistently outperform local models on MTAT and MSD, with a more pronounced performance increment on MSD.

A. Comparison of local and global models

B. Effect of multi-level features

Concatenating features across levels improves AUC on both datasets, while combining three input scales gives the best multi-scale result on both datasets.

B. Effect of multi-level features

  • Concatenating multi-level features consistently raises AUC on both MTAT and MSD relative to single-level features.Source: B. Effect of multi-level features · p. 3
  • The most important feature level differs by dataset: removing L1 causes the largest MTAT drop, while removing L3 causes the largest MSD drop.The authors suggest this may reflect MTAT's greater share of instrument-related tags relative to genre or mood tags.Source: B. Effect of multi-level features · p. 3

C. Effect of multi-scaled features

D. Performances visualization of individual tags

The section reports that the best multi-level and multi-scale features work well across nearly all tags, while tag preferences vary by dataset and tag. It also describes transfer-learning results and comparisons with prior state-of-the-art models.

D. Performances visualization of individual tags

  • Best AUCs for almost all tags occur with concatenated multi-level and multi-scale features, although individual tags favor different layers and input scales.Blues, chill, guitar, and 80s favor L2 over L3; some tags favor 54 frames, while others favor 18 frames.Source: D. Performances visualization of… · p. 4

E. Transfer learning and comparison to state-of-the-arts

  • The proposed architecture is reported as highly effective against prior state-of-the-art models, with fault-filtered GTZAN accuracy exceeding previously reported results.The passage also reports that MSD-pretrained features yield higher MTAT auto-tagging performance than local models trained on MTAT.Source: E. Transfer learning and comparison to… · p. 4

V. CONCLUSION AND FUTURE WORK

The conclusion presents an architecture designed for music tags with different abstraction levels and reports best results on three popular datasets. Future work is to optimize the local CNNs and global aggregated networks simultaneously.

V. CONCLUSION AND FUTURE WORK

  • Table II compares the authors’ models with prior state-of-the-art results.The caption identifies the comparison’s scope but supplies no numerical values.Source: Table II · p. 4
  • The authors report that the proposed architecture achieves the best results on three popular datasets.The conclusion says its effectiveness was evaluated across multi-level and multi-scale feature combinations and transfer-learning settings.Source: §V · p. 4
  • Future work is to train the architecture with multi-task learning by optimizing local CNNs and global aggregated networks simultaneously.Source: §V · p. 4
Loading 1703.01793v2…