PaperlayerView exact revision on arXiv

Source-linked AI summary

Contrastive Learning of Musical Representations

Janne Spijkervet, John Ashley Burgoyne

arXiv:2103.09410v2cs.SDcs.LGeess.AS

TL;DR

Labeled music datasets are costly to create, motivating methods that learn useful representations from abundant unlabeled audio. CLMR adapts contrastive self-supervised learning to raw musical waveforms with task-specific augmentations, and achieves competitive classification, transfer, and label-efficient results.

  • Problem

    Labeled music corpora are difficult, expensive, and time-consuming to create, while effective unsupervised representation learning for raw musical audio remains elusive.

  • Method

    CLMR combines SimCLR-inspired contrastive learning with a stochastic chain of musical-audio augmentations to learn representations from raw waveforms without labels.

  • Results

    CLMR achieves competitive performance against supervised models, comparable performance with 1% of labeled data, and out-of-domain transferability of learned representations.

  • Takeaways & Limitations

    CLMR supports task-agnostic pre-training on large raw-music datasets and data-efficient downstream music classification.

Abstract

from arXiv · show

While deep learning has enabled great advances in many areas of music, labeled music datasets remain especially hard, expensive, and time-consuming to create. In this work, we introduce SimCLR to the music domain and contribute a large chain of audio data augmentations to form a simple framework for self-supervised, contrastive learning of musical representations: CLMR. This approach works on raw time-domain music data and requires no labels to learn useful representations. We evaluate CLMR in the downstream task of music classification on the MagnaTagATune and Million Song datasets and present an ablation study to test which of our music-related innovations over SimCLR are most effective. A linear classifier trained on the proposed representations achieves a higher average precision than supervised models on the MagnaTagATune dataset, and performs comparably on the Million Song dataset. Moreover, we show that CLMR's representations are transferable using out-of-domain datasets, indicating that our method has strong generalisability in music classification. Lastly, we show that the proposed method allows data-efficient learning on smaller labeled datasets: we achieve an average precision of 33.1% despite using only 259 labeled songs in the MagnaTagATune dataset (1% of the full dataset) during linear evaluation. To foster reproducibility and future research on self-supervised learning in music, we publicly release the pre-trained models and the source code of all experiments of this paper.

1. INTRODUCTION

CLMR addresses the difficulty of obtaining labeled musical data by learning representations directly from raw, unlabeled audio through self-supervised contrastive learning. It achieves competitive classification performance, data-efficient learning, and out-of-domain transfer.

  • Labeled music corpora are difficult, expensive, and time-consuming to create, despite the abundance of raw unlabeled music data.
  • Effective unsupervised representation learning for raw musical audio has remained elusive compared with supervised learning and prior transformation-based approaches.
  • CLMR combines SimCLR-inspired contrastive learning with a chain of musical-audio augmentations to learn representations from raw waveforms without labels.
  • CLMR achieves strong music-classification performance against supervised models while using self-supervised pre-training and a linear downstream classifier.
  • CLMR achieves comparable classification performance using 1% of the labeled data and transfers representations across out-of-domain musical-audio datasets.
  • The study includes an ablation of individual audio augmentations to assess their effectiveness.

2. RELATED WORK

Prior work established self-supervised learning across domains and introduced several audio applications, but music representation learning remained comparatively limited and often depended on transformations or specialized settings.

  • Self-supervised learning has advanced across many domains, including representation learning approaches that support downstream tasks with less labeled data.
  • Audio self-supervised learning has included contrastive predictive coding, speech-oriented workers, augmentation-based transfer methods, pitch estimation, and audio-visual correspondence.
  • Compared with vision, self-supervised learning in audio remains limited, motivating further methods for musical representation learning.

3. METHOD

CLMR adapts contrastive learning to raw musical waveforms by creating correlated augmented views, encoding them, projecting them into a latent space, and optimizing a contrastive loss. Its pipeline uses stochastic audio augmentations, batch negatives, and linear evaluation of frozen representations.

  • 3. METHOD: CLMR adapts SimCLR to raw music audio, relying on the composition of multiple design components rather than one isolated choice.
  • 3. METHOD: The framework creates two independently augmented views of one audio fragment, encodes them, projects them into a latent space, and contrasts the positive pair against batch negatives.
  • 3.1 Data Augmentations: Augmentations are stochastically applied independently, with operations including random cropping, polarity inversion, noise, gain changes, filtering, delay, pitch shifting, and reverb.
  • Larger batches provide more negative examples, making the objective harder while potentially improving model performance.
  • 3.4 Contrastive Loss Function: The contrastive objective uses normalized temperature-scaled cross-entropy with cosine similarity, and computes the loss in both directions for each positive pair.
  • Representations are evaluated by freezing the pre-trained network and training a linear classifier on the downstream music-classification task.

4. EXPERIMENTAL RESULTS

CLMR representations are evaluated on multi-label music tagging, augmentation choices, label efficiency, and transfer learning. The experiments report strong MagnaTagATune performance, substantial gains with limited labels, and transferable representations across datasets.

  • Music classification: CLMR exceeds the supervised MagnaTagATune benchmark with 35.6% PR-AUC using self-supervised pre-training and a linear classifier.Adding a hidden layer raises performance by 0.4% PR-AUC, while a larger configuration reaches 37.0% PR-AUC.
  • Music classification: The Million Song Dataset performance is lower than the supervised benchmark and current state-of-the-art model.
  • Data augmentations: Adding a frequency filter improves downstream performance more substantially than gain or delay, while random cropping alone reaches 30.5 PR-AUC.
  • Data augmentations: For Polarity and Filter transformations, applying augmentations with probability p_t = 0.8 is beneficial.The study varies augmentation probability across 0, 0.4, and 0.8 to avoid making the contrastive task too hard or too simple.
  • Data efficient classification: Using 259 labeled songs, CLMR achieves 33.1% PR-AUC versus 24.8% for an equivalent supervised model trained on about 25,000 songs.This corresponds to using 100 times fewer labeled songs, or 1% of the labels, while maintaining similar performance.
  • Transfer learning: CLMR and CPC learn effective representations from out-of-domain datasets, exceeding previous supervised raw-audio systems and working on smaller GTZAN and Billboard datasets.The transfer experiments use frozen encoders followed by linear evaluation.

5. CONCLUSION

The paper presents CLMR as a label-free framework for learning useful raw-waveform music representations. Its representations achieve competitive classification, work with far fewer labels, and transfer across music datasets.

  • CLMR learns task-agnostic representations from raw musical waveforms without preprocessing or ground-truth labels.
  • CLMR achieves competitive performance with fully supervised models on MagnaTagATune and Million Song classification.
  • CLMR achieves comparable performance using 100 times fewer labeled songs and transfers across different music datasets.
  • The authors release pre-trained models and experiment source code to support reproducibility and future research.

Supplementary Material for Contrastive Learning of Musical Representations

The supplementary preprocessing pipeline converts source audio to PCM waveforms at selected sample rates. Resampling and decoding with ffmpeg are the only preprocessing steps performed before training.

  • Audio preprocessing: The experiments use raw waveform data during both pre-training and linear evaluation.
  • Audio preprocessing: MagnaTagATune fragments are reconstructed into single-song files when fragments originate from the same song.
  • Audio preprocessing: Million Song audio consists of stereo 30-second MP3 fragments sampled at 44,100 Hz.
  • Audio preprocessing: All files are resampled to 22,000 Hz, 16,000 Hz, or 8,000 Hz and decoded to PCM with ffmpeg.
  • Audio preprocessing: Resampling and decoding are the only preprocessing steps performed before training.

B. DATA AUGMENTATION DETAILS

CLMR applies a configurable chain of waveform augmentations, including cropping, polarity inversion, noise, gain, filtering, delay, pitch shifting, and reverberation. These operations alter signal properties while preserving raw-audio training inputs.

  • Augmentation pipeline: The default pre-training configuration uses eight independently probabilistic audio augmentations.Each augmentation probability is tuned by hyperparameter grid search.
  • Cropping: Audio is cropped to 20,736, 43,740, or 59,049 samples for sample rates of 8,000, 16,000, and 22,050 Hz.
  • Signal perturbations: The pipeline includes polarity inversion, white Gaussian noise at 80 dB SNR, and random gain reduction between -6 and 0 dB.
  • Filtering: Filtering randomly selects low-pass or high-pass processing, with cutoff frequencies drawn from separate frequency ranges.Low-pass cutoffs range from 2,200 to 4,000 Hz, while high-pass cutoffs range from 200 to 1,200 Hz.
  • Delay: Delay adds a randomly selected 200–500 ms echo in 50 ms increments at volume factor 0.5.
  • Pitch and reverberation: Pitch shifting changes the signal by a uniformly sampled interval from -5 to 5 semitones, and reverberation applies a Schroeder effect.

C. ADDITIONAL EXPERIMENTAL RESULTS

Larger contrastive-learning batch sizes improve downstream linear music-classification performance, although they increase the complexity of the pretext objective.

  • Larger batch sizes increased linear music-classification performance during self-supervised pre-training.The smallest model was already competitive with fully supervised models, while performance increased at 96 examples per batch.

C.2 Sample Rates

Downsampling the input audio to 8,000 or 16,000 Hz causes only a marginal penalty in the final self-supervised-model scores.

  • A marginal penalty occurs when self-supervised models use 8,000 Hz or 16,000 Hz audio sampling rates.The experiment isolates resampling by applying only random cropping, because resampling disturbs the frequency spectrum.

C.3 Additional Hidden Layer and Training Duration

Longer self-supervised pre-training improves downstream music-classification performance, while evaluation compares linear classifiers with classifiers augmented by one hidden layer.

  • Additional hidden layer: The experiment compares linear evaluation with adding a single hidden layer to the downstream classifier.Table C.3 reports linear-evaluation scores to the left of each slash and hidden-layer scores to the right, across different pre-training durations.
  • Training duration: Increasing self-supervised training duration improves downstream music-classification performance.The models are pre-trained from scratch until convergence with a batch size of 96.

C.4 Qualitative Results

Qualitative analyses show that self-supervised representations separate music classes and learn structured frequency responses, while tag-wise performance remains close to the supervised benchmark.

  • Class manifolds: Both CLMR and CPC representations cleanly separate music-track classes in t-SNE visualisations.The visualisations use fragments from randomly selected validation-set tracks.
  • Learned filters: CLMR’s first convolutional layer responds to a narrow band around 7,500 Hz, while higher-layer filters span the full frequency range.Higher-layer filters spread first linearly and then non-linearly across the frequency range.
  • Tag-wise classification: No top-50 MagnaTagATune tag loses more than 4% ROC-AUC under self-supervised pre-training with linear evaluation versus the supervised benchmark.The comparison covers CLMR, CPC, and a fully end-to-end-trained supervised model.
Loading 2103.09410v2…