MERT: Acoustic Music Understanding Model with Large-Scale Self-supervised Training

Yizhi Li; Ruibin Yuan; Ge Zhang; Yinghao Ma; Xingran Chen; Hanzhi Yin; Chenghao Xiao; Chenghua Lin; Anton Ragni; Emmanouil Benetos; Norbert Gyenge; Roger Dannenberg; Ruibo Liu; Wenhu Chen; Gus Xia; Yemin Shi; Wenhao Huang; Zili Wang; Yike Guo; Jie Fu

Source-linked AI summary

MERT: Acoustic Music Understanding Model with Large-Scale Self-supervised Training

Yizhi Li, Ruibin Yuan, Ge Zhang, Yinghao Ma, Xingran Chen, Hanzhi Yin, Chenghao Xiao, Chenghua Lin, Anton Ragni, Emmanouil Benetos, Norbert Gyenge, Roger Dannenberg, Ruibo Liu, Wenhu Chen, Gus Xia, Yemin Shi, Wenhao Huang, Zili Wang, Yike Guo, Jie Fu

arXiv:2306.00107v5May 31, 2023cs.SDcs.AIcs.CLcs.LGeess.AS

TL;DR

Music audio SSL lacks a general-purpose, cost-effective open-source model, partly because musical tonal and pitched characteristics are difficult to model. MERT addresses this with acoustic and musical teacher models in MLM-style pre-training, then scales the approach from 95M to 330M parameters. It generalises across 14 music understanding tasks and achieves SOTA or comparable overall performance with substantially fewer parameters than previous self-supervised baselines.

Problem
Music audio SSL remains insufficiently explored, with limited general-purpose, cost-effective open-source models and distinctive challenges in modelling tonal and pitched characteristics.
Method
MERT uses MLM-style acoustic pre-training with RVQ-VAE acoustic and CQT musical teachers, alongside strategies designed to improve training stability and scaling.
Results
MERT generalises across 14 music understanding tasks, achieves SOTA or comparable overall performance, and uses substantially fewer parameters than previous self-supervised baselines.
Takeaways & Limitations
MERT provides an open-source, generalisable, and computationally affordable acoustic music pre-trained model for music understanding tasks.
Takeaways & Limitations
Training on 5-second signals may constrain tasks requiring extended musical context, while larger half-precision settings still experience gradient-explosion issues.

Abstract

from arXiv · show

Self-supervised learning (SSL) has recently emerged as a promising paradigm for training generalisable models on large-scale data in the fields of vision, text, and speech. Although SSL has been proven effective in speech and audio, its application to music audio has yet to be thoroughly explored. This is partially due to the distinctive challenges associated with modelling musical knowledge, particularly tonal and pitched characteristics of music. To address this research gap, we propose an acoustic Music undERstanding model with large-scale self-supervised Training (MERT), which incorporates teacher models to provide pseudo labels in the masked language modelling (MLM) style acoustic pre-training. In our exploration, we identified an effective combination of teacher models, which outperforms conventional speech and audio approaches in terms of performance. This combination includes an acoustic teacher based on Residual Vector Quantisation - Variational AutoEncoder (RVQ-VAE) and a musical teacher based on the Constant-Q Transform (CQT). Furthermore, we explore a wide range of settings to overcome the instability in acoustic language model pre-training, which allows our designed paradigm to scale from 95M to 330M parameters. Experimental results indicate that our model can generalise and perform well on 14 music understanding tasks and attain state-of-the-art (SOTA) overall scores.

1 INTRODUCTION

MERT addresses the lack of a general-purpose, cost-effective open-source model for acoustic music understanding with a scalable self-supervised pre-training framework. Its 330M-parameter model achieves overall SOTA results across music information retrieval tasks while remaining much smaller than Jukebox.

Existing acoustic music studies largely target music tagging and often lack open-source code or checkpoints for further evaluation.
JukeMIR provides broad MIR evaluation but uses billion-parameter hierarchical autoregressive decoders, making general music understanding computationally inefficient.Inference can take weeks on datasets such as MTG using a consumer-grade 3090 GPU.
MERT uses teacher-generated pseudo targets in a multi-task self-supervised paradigm to balance acoustic and musical representation learning.The framework is designed to capture music's pitched and tonal characteristics.
330M parameters enable MERT to achieve overall SOTA results across various MIR tasks, while scaling from 95M to 330M through strategies addressing acoustic pre-training instability.The 330M model is only 7% the size of Jukebox.
The contributions include a multi-task predictive acoustic SSL paradigm, broad ablations, stable scaling strategies, and an open-source, generalisable, computationally affordable model.

2 RELATED WORK

Related work positions acoustic music pre-trained models as reusable representations for MIR, while highlighting unresolved needs for open-source, generalisable, lightweight models. The paper also connects music processing to speech and language-modeling approaches.

PLMs for Acoustic Music: Music information retrieval faces data-availability challenges because music annotation is costly and subject to country-specific copyright laws.
PLMs for Acoustic Music: Acoustic music pre-trained models enable transfer learning across downstream MIR tasks without requiring extensive data annotation.
PLMs for Acoustic Music: Existing acoustic music pre-trained models still need more open-source, generalisable, and lightweight representations for industrial and research applications.
Self-Supervised Speech Processing: Shared audio formats and the cocktail party problem motivate adapting speech pre-trained models to acoustic music processing.
Audio Representation with Language Modelling: Mask-strategy language models remain under-explored for acoustic music, while prior audio work uses extracted discrete tokens from continuous features.

3 METHODOLOGY

MERT adapts masked-language-model-style self-supervised pre-training to music by predicting acoustic pseudo-labels and reconstructing musical features. Its methodology combines music-specific teachers, augmentation, and training strategies to model acoustic, timbral, harmonic, and pitch information.

3.1 PRE-TRAINING WITH MLM: MERT adapts HuBERT-style masked prediction by learning pseudo-labels from masked sequential audio inputs.The pre-training paradigm predicts acoustic teachers and reconstructs music teachers using masked language modelling.
3.2 MODELLING ACOUSTIC INFORMATION: The alternative acoustic teacher clusters log-Mel and Chroma features into 60,000 combined classes to represent timbre and harmonic information.The configuration uses 300-means for the log-Mel spectrum and 200-means for Chroma features, while retaining HuBERT-comparable complexity.
3.2 MODELLING ACOUSTIC INFORMATION: EnCodec supplies discrete acoustic targets through eight residual VQ-VAE codebooks that preserve waveform timbre information.For 5-second, 24-kHz waveforms, EnCodec produces 375 frames across 8 deep acoustic features at 75 Hz.
3.3 MODELLING MUSICAL INFORMATION: The model uses a multi-task objective that combines acoustic prediction with CQT-spectrum reconstruction for musical-pitch information.The final loss is a linear combination of the acoustic loss LH and musical-pitch loss LCQT, with CQT reconstruction trained using MSE.
3.3 MODELLING MUSICAL INFORMATION: CQT reconstruction emphasizes pitch because its frequency bins are proportional to frequency, assigning each octave the same number of bins.The paper uses CQT because it offers a better time-frequency trade-off for music with multiple pitches across octaves.

4 EXPERIMENTS

MERT is evaluated on 14 downstream music-understanding tasks using frozen-backbone probing, broad baselines, and multiple training scales. The experiments also examine training instability and report table-based comparisons across models and tasks.

Downstream Tasks: 14 downstream tasks cover frame-level music understanding and sequential applications, including tagging, key detection, pitch classification, beat tracking, and source separation.Instrument classification uses Nsynth and MTG-instrument datasets with ROC and average precision metrics.
Evaluation Protocol: Frozen-backbone probing evaluates pretrained models as deep feature extractors while training only a simple downstream structure, typically an MLP.The protocol also limits hyperparameter search for fair comparison.
Baseline Methods: Baselines span supervised music tagging, contrastive music representations, generative musical representations, and speech SSL models.The compared models include MusiCNN, CLMR, MULE, Jukebox, JukeMIR, HuBERT, and data2vec.
Training Settings: 160K hours of Internet-mined music recordings support matched-scale training, with 1K hours for 95M models and the full dataset for the 330M model.A public 95M edition uses only an open-source music dataset.
Training Stability: Scaling acoustic encoder-only models exposes greater training instability, including gradient clipping, loss spikes, impaired MLM accuracy, and downstream degradation.DeepNorm was ineffective; attention relaxation helped address instability, while Pre-LN offered a potential way to continue training.
Experimental Results: Tables 1 and 2 compare MERT with supervised and unsupervised baselines across downstream tasks, including average scores across applicable models and SOTA results.Table 1 identifies acoustic-teacher categories and marks public-data training and reference-claimed results.

5 RESULTS ANALYSIS

MERT combines acoustic and musical teacher signals to produce compact, generalisable representations across music understanding tasks. Results show strong performance with limited data, while CQT-based musical supervision improves tasks requiring pitch and higher-level musical information.

5.1 PERFORMANCE & EFFICIENCY OF MERT MODELS: MERT-330MRVQ-VAE matches the average score of previous SOTAs and becomes the new SOTA on 4 metrics.MERT-95M models retain comparable performance, while MERTs perform especially well on beat, pitch, and local timbre tasks and remain competitive on global tasks.
5.1 PERFORMANCE & EFFICIENCY OF MERT MODELS: MERT-95M-public and MERT-95M achieve competitive performance against SOTA and MERT-330M using a ∼1k hour training dataset.MERT-95M-public uses 910 hours of mainly pop music and differs significantly from MERT-95M only on GTZAN genre classification.
5.1 PERFORMANCE & EFFICIENCY OF MERT MODELS: 95M and 330M MERT models use only 1.9% and 6.6% of Jukebox-5B’s parameters while achieving better or comparable performance.Scaling to 330M improves or maintains performance on most tasks, with beat tracking as the exception.
5.2 THE EFFECTIVENESS OF ACOUSTIC & MUSICAL TEACHER: Adding Chroma or CQT features raises key-detection accuracy from 15.6 to 55.1 and 53.2, respectively, while maintaining or improving other task performance.These results indicate that additional pitch-related pseudo-targets provide useful inductive bias for acoustic pre-training.
5.2 THE EFFECTIVENESS OF ACOUSTIC & MUSICAL TEACHER: CQT musical teachers improve key detection, genre classification, and emotion recognition compared with acoustic-only or Chroma-augmented teachers.The CQT teacher introduces harmonic information that is aligned with musical audio.
5.2 THE EFFECTIVENESS OF ACOUSTIC & MUSICAL TEACHER: EnCodec RVQ-VAE is selected as the final acoustic teacher because K-means is difficult to scale on large datasets.The intermediate discrete codecs can support recovery of audio through a neural decoder, providing comprehensive acoustic information.

6 CONCLUSION

MERT integrates acoustic and musical teacher models for masked language model pre-training, learning robust music representations with improved stability. It achieves SOTA or comparable results across many MIR tasks while using substantially fewer parameters than previous self-supervised baselines.

MERT combines RVQ-VAE and CQT teacher models to provide acoustic and musical information for MLM-based music understanding pre-training.In-batch noise mixup and Pre-LN further support robust representations and training stability.
MERT achieves SOTA or comparable performance across a wide range of MIR tasks while using a significantly smaller parameter size.

LIMITATIONS AND FUTURE WORK

MERT is trained on 5-second audio because of computational and sequence-length constraints, which may limit performance on tasks requiring extended musical context. Larger models and batches also remain vulnerable to gradient-explosion issues in half-precision training, while inverse scaling is observed in some settings.

5-second training signals may limit performance on tasks requiring comprehensive understanding of extended musical contexts.The models can handle longer sequences through relative positional embeddings, but the authors plan longer-context training with additional computing resources.
Larger model sizes and batch sizes still suffer gradient-explosion issues during half-precision acoustic pre-training.
The authors also observe an inverse-scaling effect in some settings.

A.1 ROBUST REPRESENTATION LEARNING

MERT introduces in-batch noise mixup and evaluates pre-training settings for robust music representations. The supplementary evaluation finds that vanilla EnCodec RVQ-VAE continuous representations alone are insufficient as a robust baseline.

Noise mixup: In-batch noise mixup adds randomly selected excerpts from the same batch to audio clips during pre-training, creating augmented samples instead of using the originals.The authors motivate this augmentation as improving robustness by encouraging attention to useful musical sources while ignoring noise.
Noise mixup: A mixup probability of 0.5 was selected for the released model because mixup worsened MERTK-means but improved MERTRVQ-VAE performance on average.The probability was chosen using the average performance score under the evaluation setting with musical loss.
RVQ-VAE representations: Vanilla continuous representations from the EnCodec RVQ-VAE encoder alone are insufficient for a robust music understanding baseline.This comparison was conducted to verify whether improvements arise from MERT pre-training rather than the continuous representations alone.

B.1 DOWNSTREAM TASKS

The evaluation covers 14 downstream music information retrieval tasks spanning tagging, tonal, rhythmic, source-separation, vocal, instrumental, and emotional understanding. Pre-trained representations are frozen and evaluated as fixed-depth embeddings with lightweight task-specific heads.

Benchmark scope: The benchmark evaluates models on 14 downstream tasks to provide a comprehensive view of generalisation and baseline comparisons.The tasks include music tagging, key and pitch analysis, beat tracking, source separation, instrument and singer identification, vocal technique detection, and emotion regression.
Task definitions: Key detection predicts a song’s tonal scale and dominant pitch level using Giantsteps evaluation data and refined accuracy with error tolerance.Training and validation use a commonly used Giantsteps-MTG-keys subset with the split from prior work.
Task definitions: Beat tracking produces frame-level beat predictions, post-processes them with a dynamic Bayesian network, and evaluates timing agreement using the f measure.A prediction is correct when its event differs from ground truth by no more than 20ms.
Task definitions: Source separation demixes recordings into vocals, drums, bass, and other stems using MUSDB18 with the official 86/14/50 train-validation-evaluation split.The benchmark contains 150 full-length tracks and multiple isolated stems.
Evaluation protocol: Pre-trained model parameters are frozen, and fixed-depth embeddings are fed to lightweight downstream prediction heads to test representation reusability.Most tasks use a one-hidden-layer MLP, while source separation uses the specified three-layer LSTM post-processing baseline.

B.3 TRAINING INSTABILITY

Scaling MERT to 330M parameters exposed training instability in several Transformer configurations. Attention relaxation ultimately enabled stable training beyond 100K steps, while representation visualisations showed genre-related clustering without consistently stronger clustering in higher layers.

Training stability: 330M-model experiments monitored gradient norm, loss scale, and acoustic and musical MLM losses under mixed-precision training.These training curves are reported in Figure 2, which displays the acoustic MLM loss for RVQ-VAE codebook 0.
Training stability: Vanilla Pre-LN training crashed around 50K steps, and a restart with gradient clipping reduced from 10.0 to 1.0 crashed again after 40K steps.The second run failed after reaching the minimum loss scale.
Training stability: DeepNorm-style initialization and enhanced residual connections caused model collapse around 20K steps rather than resolving instability.The authors suspected increased Transformer depth could contribute to the instability.
Training stability: Attention relaxation alleviated the overflow problem and allowed the final MERT-330M model to train stably for over 100K steps.The method adds a scale constant to the softmax calculation in the attention module.
Representation visualisation: Both MERT-95M-publicK-means and MERT-330MRVQ-VAE representations clustered by genre across visualised layers, but higher layers did not necessarily cluster more strongly.The authors suggest genre may not be the most abstractive label or that top layers focus more on MLM objectives.