PaperlayerView exact revision on arXiv

Source-linked AI summary

Multi-label Music Genre Classification from Audio, Text, and Images Using Deep Features

Sergio Oramas, Oriol Nieto, Francesco Barbieri, Xavier Serra

arXiv:1707.04916v1cs.IR

TL;DR

Music-genre classification is commonly framed as assigning one broad label, despite overlapping and finer-grained categories. This paper presents a multimodal dataset and deep-learning approach for multi-label classification. Its reported results favor deep features over handcrafted approaches and show that combining modalities can improve results.

  • Problem

    Most related work assigns one broad genre label, although genre categories can overlap and more specific genres exist.

    Source: §1 · p. 1

  • Method

    The paper introduces MuMu, with audio, text reviews, and cover images, and applies deep-learning feature embeddings to multi-label genre classification across these modalities.

    Source: §1 · p. 1

  • Results

    Deep feature learning surpasses handcrafted approaches, and late fusion scores better than each modality alone.

    Source: §1 · p. 1

  • Takeaways & Limitations

    Text approaches seem to outperform other modalities, while image classification has the lowest performance but helps when combined with other modalities.

    Source: §7 · p. 6

    The paper describes itself as an initial attempt and identifies image-based latent-factor learning and applying the multimodal approach to other MIR tasks as future work.

    Source: §7 · p. 6

Abstract

from arXiv · show

Music genres allow to categorize musical items that share common characteristics. Although these categories are not mutually exclusive, most related research is traditionally focused on classifying tracks into a single class. Furthermore, these categories (e.g., Pop, Rock) tend to be too broad for certain applications. In this work we aim to expand this task by categorizing musical items into multiple and fine-grained labels, using three different data modalities: audio, text, and images. To this end we present MuMu, a new dataset of more than 31k albums classified into 250 genre classes. For every album we have collected the cover image, text reviews, and audio tracks. Additionally, we propose an approach for multi-label genre classification based on the combination of feature embeddings learned with state-of-the-art deep learning methodologies. Experiments show major differences between modalities, which not only introduce new baselines for multi-label genre classification, but also suggest that combining them yields improved results.

1. INTRODUCTION

The paper reframes music-genre classification around multiple, fine-grained labels and introduces MuMu, a multimodal dataset for studying audio, text, and image features. Its results report advantages for deep features and late fusion over individual modalities.

1. INTRODUCTION

  • The paper targets multi-label classification of fine-grained genres because broad genres such as Pop and Rock can miss more specific categories.Genre categories may overlap, so a track can combine Pop with Deep House or Reggae elements.Source: §1 · p. 1
  • MuMu contains roughly 31k albums across 250 genres, with cover images, text reviews, and audio tracks for multimodal classification.The passage reports approximately 147k audio tracks and 447k album reviews.Source: §1 · p. 1
  • Deep feature learning surpasses handcrafted-feature approaches, narrows the text–audio classification gap, and late fusion scores better than each modality alone.The study also reports improved results from dimensionality reduction in audio classification.Source: §1 · p. 1

2. RELATED WORK

Prior music-genre work mainly uses audio and broad single-class classification, while text, image, and multimodal approaches are less common. Multi-label genre work is scarce despite a longer MIR tradition of multi-label tag classification.

2. RELATED WORK

  • Audio is the dominant source, with traditional systems using handcrafted features and newer systems feeding spectrogram representations to CNNs.Text approaches use album reviews or lyrics, while the passage notes relatively few image-based genre-classification papers.Source: §2 · p. 1
  • Existing multimodal genre work combines audio with lyrics or video, but the authors knew of no published deep-learning multimodal approach for this task.Source: §2 · p. 2
  • Multi-label genre classification has few dedicated approaches, although multi-label tag classification has a long history in MIR.Source: §2 · p. 2

3. MULTIMODAL DATASET

MuMu combines album-level audio, text, image, and multi-label annotations, addressing the stated lack of a publicly available large-scale dataset with all four. Its hierarchical genre labels are imbalanced and span multiple taxonomy levels.

3. MULTIMODAL DATASET

  • MuMu combines Amazon Reviews and Million Song Dataset information because the authors knew of no public large-scale dataset spanning audio, images, text, and multi-label annotations.Source: §3 · p. 2
  • Entity resolution mapped 60% of Amazon albums; filtering to recordings linked to a unique album yielded 147,295 songs across 31,471 albums.Source: §3 · p. 2
  • The released dataset includes mappings, genre annotations, splits, reviews, and image links, but audio and image files are withheld because of copyright issues.Source: §3 · p. 2

3.1 Genre Labels

  • The dataset retains 250 genres annotated in at least 12 albums, propagates hierarchical branch labels, and has highly unbalanced genre frequencies.Albums can receive labels at multiple taxonomy levels; the passage describes the imbalance as a classification challenge.Source: §3.1 · p. 2

4. MULTI-LABEL CLASSIFICATION

The paper contrasts independent-label logistic prediction with a factorized representation that captures label relationships. It evaluates label-wise discrimination and the diversity of predicted labels.

4. MULTI-LABEL CLASSIFICATION

  • The LOGISTIC approach uses sigmoid outputs and cross-entropy but assumes statistical independence between classes, which the passage says does not hold for music genres.Source: §4 · pp. 2–3

4.1 Labels Factorization

  • PPMI label co-occurrences are factorized with SVD into lower-dimensional label factors, which are multiplied by the item-label matrix to obtain item factors in the same space.Source: §4.1 · p. 3

4.2 Evaluation Metrics

  • For probabilistic multi-label classification, AUC is computed for every label and averaged; catalog coverage@k measures the percentage of normalized unique labels in an algorithm’s top-k predictions across all test items.Source: §4.2 · p. 3

5. ALBUM GENRE CLASSIFICATION

The section frames genre prediction at the album level and describes separate audio, text, and image models whose learned feature vectors are combined for multimodal classification.

5. ALBUM GENRE CLASSIFICATION

  • Album genres are predicted from album-level audio, text reviews, and cover images, with each modality represented separately.The section focuses on albums because each modality is associated with an album.Source: §5 · p. 3

5.1 Audio-based Approach

  • Audio features are learned from tracks, averaged across each album, and used to predict album genres in a shallow network.The CNN receives fixed-size, 15-second log-compressed CQT spectrogram patches.Source: §5.1 · p. 3

5.2 Text-based Approach

  • Album reviews are aggregated into text capped at 1,000 characters, then represented with tf-idf VSM features, optionally enriched with Wikipedia categories.Both text representations use vocabularies limited to 10,000 terms and feed a two-layer dense network.Source: §5.2 · p. 4

5.3 Image-based Approach

  • Cover images are classified with an ImageNet-pretrained ResNet-101 fine-tuned using a sigmoid logistic layer and binary cross-entropy loss.The network uses image augmentation and weight decay.Source: §5.3 · p. 4

5.4 Multimodal approach

  • Instead of jointly training modalities, the approach trains each separately, then L2-regularizes and concatenates their learned album vectors for an MLP.Separate training is motivated by concern that a stronger modality could dominate joint learning; the MLP output can be LOGISTIC or COSINE.Source: §5.4 · p. 4

6. EXPERIMENTS

Experiments compare audio, text, image, and combined systems on MuMu. CNNs outperform handcrafted audio features, text models slightly exceed audio, image classification trails the other modalities, and combining modalities—especially all three—performs best.

6. EXPERIMENTS

  • The evaluation uses an 80/10/10 train-validation-test split, compares individual modalities, then combines the best-AUC feature vector from each modality.Reported measures include AUC, Coverage@k for k = 1, 3, 5, epoch training time, and network hyperparameters.Source: §6 · p. 4
  • A t-SNE visualization of d = 50 album factors shows clustering by genre among albums with a single top-level label.The factors come from factorizing the training and validation album-annotation matrix.Source: §6 · p. 4, Figure 1 · p. 5

6.4 Multimodal Classification

  • Combining modalities outperforms single-modality approaches, and the best approaches use audio, text, and image features together.Text features tend to improve results; image features appear to help multimodal LOGISTIC approaches, while COSINE combinations have similar AUC but better catalog coverage than LOGISTIC.Source: Table 2 · p. 5, §6.4 · p. 6

6.1 Audio Classification

  • TIMBRE-MLP achieves 0.792 AUC, compared with 0.888 AUC for the best CNN approach.The CNNs applied to audio spectrograms outperform traditional handcrafted-feature approaches.Source: §6.1 · p. 5
  • COSINE configurations achieve better AUC in most tested audio settings, while filter width and shape trade off performance, parameter count, and training time.Time-only 4X96 networks are faster and comparable in performance; 4X70 networks achieve better results with slightly higher parameter counts and training time than 4X96.Source: §6.1 · p. 5

6.2 Text Classification

  • Semantic text enrichment improves AUC and diversity, COSINE slightly improves AUC and greatly improves coverage over LOGISTIC, and text results slightly exceed audio overall.Informative words vary by genre, including band for Rock, song for Pop, and place names such as Jamaica for Reggae.Source: §6.2 · p. 6

6.3 Image Classification

  • Image classification has lower AUC and catalog coverage than the other modalities; cover-image features also show genre-associated visual patterns.The reported image results use LOGISTIC, and the authors note patterns such as foreground women’s faces in Country covers and clustered Jazz covers.Source: §6.3 · p. 6

7. CONCLUSIONS

The study applies deep-learning multi-label genre classification across audio, text, images, and their combinations; multimodal methods appear to outperform single modalities, with all three modalities together achieving the best results. It is an initial study, with image latent-factor learning and extension to other MIR tasks left for future work.

7. CONCLUSIONS

  • Multimodal approaches appear to outperform single-modality approaches, and aggregating audio, text, and images achieves the best results.Source: §7 · p. 6
  • Text-based approaches seem to outperform other modalities and benefit from entity linking, while image-based classification yields the lowest performance but helps in combination.Source: §7 · p. 6
  • Reducing target-label dimensionality improves both accuracy and catalog coverage.Source: §7 · p. 6
  • The paper frames its findings as an initial study and leaves image latent-factor learning and application to other MIR tasks for future work.Source: §7 · p. 6
Loading 1707.04916v1…