PaperlayerView exact revision on arXiv

Source-linked AI summary

MuQ: Self-Supervised Music Representation Learning with Mel Residual Vector Quantization

Haina Zhu, Yizhi Zhou, Hangting Chen, Jianwei Yu, Ziyang Ma, Rongzhi Gu, Yi Luo, Wei Tan, Xie Chen

arXiv:2501.01108v2cs.SDcs.AIcs.CLcs.LGeess.AS

TL;DR

Music SSL methods have struggled to capture both semantic and acoustic information. MuQ addresses this with Mel-RVQ targets and outperforms prior models across downstream tasks, while MuQ-MuLan reaches a ROC-AUC of 79.3 for zero-shot tagging.

  • Problem

    Previous music SSL methods struggled to capture semantic and acoustic information, including melody, chords, and tonality, simultaneously.

  • Method

    MuQ uses masked language modeling with Mel-RVQ targets that directly quantize Mel spectrograms through a lightweight linear residual vector quantization structure.

  • Results

    MuQ outperforms prior self-supervised models across diverse downstream tasks with 0.9K hours of data, while MuQ-MuLan achieves a 79.3 ROC-AUC on MagnaTagATune zero-shot tagging.

  • Takeaways & Limitations

    Mel-RVQ supports a stable and efficient target for music SSL, and MuQ-MuLan achieves state-of-the-art music-text alignment.

  • Takeaways & Limitations

    The authors acknowledge potential ethical concerns, prioritize protecting personal information and avoiding bias, and state that no audio will be distributed or shared.

Abstract

from arXiv · show

Recent years have witnessed the success of foundation models pre-trained with self-supervised learning (SSL) in various music informatics understanding tasks, including music tagging, instrument classification, key detection, and more. In this paper, we propose a self-supervised music representation learning model for music understanding. Distinguished from previous studies adopting random projection or existing neural codec, the proposed model, named MuQ, is trained to predict tokens generated by Mel Residual Vector Quantization (Mel-RVQ). Our Mel-RVQ utilizes residual linear projection structure for Mel spectrum quantization to enhance the stability and efficiency of target extraction and lead to better performance. Experiments in a large variety of downstream tasks demonstrate that MuQ outperforms previous self-supervised music representation models with only 0.9K hours of open-source pre-training data. Scaling up the data to over 160K hours and adopting iterative training consistently improve the model performance. To further validate the strength of our model, we present MuQ-MuLan, a joint music-text embedding model based on contrastive learning, which achieves state-of-the-art performance in the zero-shot music tagging task on the MagnaTagATune dataset. Code and checkpoints are open source in https://github.com/tencent-ailab/MuQ.

Introduction

The introduction identifies the difficulty of learning representations that capture music’s semantic and acoustic properties, then presents MuQ and Mel-RVQ as efficient solutions. It further highlights MuQ’s strong downstream performance and MuQ-MuLan’s music–text alignment capability.

  • Motivation: Music understanding requires capturing both semantic information and acoustic properties such as melody, chords, and tonality.Semantics-oriented SSL methods have struggled because they do not simultaneously capture both aspects.
  • Limitations of Existing Methods: Target extraction is critical in SSL because models learn by predicting tokenized pseudolabels.BEST-RQ depends strongly on random-projection initialization, whereas Encodec incurs heavy computation costs.
  • Proposed Method: MuQ addresses these limitations by using Mel-RVQ to directly quantize Mel spectrograms with a pre-trained linear residual vector quantizer.The proposed structure is designed to improve training stability and efficiency.
  • Results: MuQ demonstrates state-of-the-art performance across a wide range of downstream music understanding tasks compared with MERT and MusicFM.The introduction attributes these gains to Mel-RVQ’s enhancement of SSL performance across varied downstream tasks.
  • Results: 0.9K hours of data suffice for MuQ to outperform prior state-of-the-art SSL models, using 100x less data than comparable models require.MuQ-MuLan further aligns music and text representations through contrastive learning and achieves a ROC-AUC score of 79.3 on MagnaTagATune zero-shot music tagging.

Related Work

Prior work spans self-supervised audio representation learning, iterative refinement, and music understanding, while music-text models commonly use contrastive two-tower architectures. MuQ instead iteratively trains through its quantizer and uses MuQ as the music encoder in MuQ-MuLan.

  • Self-supervised learning for speech and audio: SSL methods use offline K-means targets, online random projection quantizers, or combined online and offline objectives.HuBERT extracts K-means labels offline, BEST-RQ generates targets online, and MT4SSL integrates both objectives.
  • Self-supervised learning for speech and audio: MuQ iteratively trains directly through the quantizer, avoiding K-means clustering.This differs from existing iterative practices based on clustering, including HuBERT and Seed-ASR.
  • Music processing and understanding: Music SSL models including CLMR, MERT, and MusicFM achieve impressive performance across multiple MIR tasks, supporting generalized music understanding.MIR evaluation includes music tagging, key detection, pitch detection, and emotion analysis.
  • Music-text joint embedding model: Music-text representation models typically use music and text encoders in a two-tower structure trained with contrastive learning loss.LAION-CLAP is open source, whereas MuLan is not open source for either its model or data.
  • Music-text joint embedding model: MuQ-MuLan follows prior music-text embedding work while replacing its music encoder with MuQ.This connects the proposed MuQ representation model to joint music-text embedding.

Method

MuQ learns music representations by predicting Mel-RVQ tokens from masked Mel spectra with Conformer-based context modeling. The method uses lightweight residual quantization, iterative refinement, and a music-text extension trained with contrastive learning.

  • MuQ self-supervised learning: MuQ masks Mel-spectrum inputs with random noise, processes them through Conformer layers, and predicts token labels using linear heads optimized by cross-entropy loss.The masking probability is denoted p.
  • Mel-RVQ tokenizer: Mel-RVQ extracts target tokens directly from Mel spectra using residual vector quantization with a single-layer linear encoder and decoder.Because Mel-RVQ produces N tokens per time step, MuQ uses N distinct linear prediction layers.
  • Mel-RVQ tokenizer: Mel-RVQ training combines codebook, commitment, and reconstruction losses in a weighted objective: loss = α · losscode + β · losscomm + lossrecon.The codebook fits embeddings to dimension-reduced features, commitment trains the projection, and reconstruction trains the decoder and codebook to restore the original feature.
  • Mel-RVQ tokenizer: The method applies residual modeling across multiple quantization steps, passing residual signals between codebooks to generate multiple tokens at each audio time step.The encoder and decoder use residual-projection matrices composed of multiple projection steps.
  • Iterative training: Iterative training fits Mel-RVQiter to latent representations extracted by an initial MuQ model, then uses its tokens to train MuQiter in a second stage.The initial stage trains Mel-RVQ on Mel spectra and the first MuQ version before latent-based refinement.
  • MuQ-MuLan: MuQ-MuLan combines a pre-trained MuQ music encoder with a text encoder in a two-tower architecture trained on paired music-text data using decoupled contrastive learning.The text encoder is based on RoBERTa with additional Transformer layers, while both modalities are pooled and projected into embeddings of dimension d.

Experiments Setup

MuQ uses a 12-layer, 310M-parameter Conformer with Mel-RVQ targets and fixed 30-second Mel-spectrum inputs. Experiments span pre-training at 0.9K and 160K hours, MuQ-MuLan contrastive training, and nine downstream tasks.

  • MuQ architecture and inputs: MuQ stacks 12 Conformer layers with 310M parameters and predicts Mel-RVQ tokens using N = 8 codebooks of size K = 1024.Audio is represented as 128-dimensional Mel spectra pooled to 25Hz before entering the Conformer.
  • MuQ architecture and inputs: The MuQ input is fixed at 30 seconds with a sampling rate of 24K Hz, and the implementation uses the Fairseq toolkit.The audio pipeline uses 128-dimensional Mel spectra pooled to 25Hz.
  • MuQ-MuLan setup: MuQ-MuLan uses a pre-trained MuQ music encoder and multilingual RoBERTa text encoder followed by 8 Transformer layers, with embedding dimension d = 512.Its 630M parameters remain unfrozen during contrastive training on 130K hours of music paired with non-personal text descriptions.
  • Pre-training setup: Pre-training uses 0.9K hours of Music4all data initially and later scales to 160K hours of in-house music.Music4all contains 0.9K hours of open access music data.
  • Pre-training setup: All MuQ pre-training runs use 32 A100-40G GPUs, batch size 192, and masking probability p = 0.6, with 75K or 200K initial steps depending on dataset scale.The 160K-hour setting adds 150K iterative-training steps after training Mel-RVQiter from the 10th-layer latent; the full process takes about 2 weeks.
  • Evaluation setup: MuQ is evaluated on nine MARBLE downstream tasks using layer-wise linear probes, while MuQ-MuLan uses zero-shot music tagging by comparing music and text embeddings.Downstream datasets are excluded from MuQ pre-training data.

Results and Analysis

MuQ outperforms MERT and MusicFM across many downstream tasks, with larger pre-training data and iterative training improving results further. Mel-RVQ’s trained residual quantization, increasing codebooks, shallow projection, and MuQ initialization strengthen performance, including state-of-the-art zero-shot tagging with MuQ-MuLan.

  • Overall results: MuQm4a, pre-trained on the 0.9Khour Music4All dataset, already outperforms MERT and MusicFM, while MuQiter achieves the best average score of 77.0.Larger pre-training data amplifies MuQ’s advantage, and iterative training slightly improves effectiveness.
  • Overall results: MuQ outperforms previous models especially on genre classification, singer identification, vocal technique detection, music structure analysis, and instrument classification.The results are visualized on the MARBLE benchmark radar plot.
  • Mel-RVQ ablation: Training the VQ instead of leaving it random boosts MusicFM and MuQ performance by 1.6 and 2.8, respectively, while Mel-RVQ’s residual structure benefits MuQ over MusicFM.The ablation models use 310M parameters, 75K Music4all pre-training steps, and five tested tasks.
  • Mel-RVQ ablation: Performance improves as the number of codebooks increases, whereas replacing the single-layer Mel-RVQ projection with a 3-layer network degrades performance.The findings support multi-target learning and indicate that a single linear projection is sufficient.
  • Tokenization targets: Mel-RVQ outperforms the Encodec tokenization target despite its lightweight structure, which the authors attribute to Encodec tokens being harder for the model to learn.The comparison appears in Table 4.
  • MuQ-MuLan: MuQ-MuLan reaches state-of-the-art zero-shot music tagging performance in both ROC-AUC and PR-AUC, outperforming prior models and retaining a slight advantage over MuLan.It also outperforms MERT-MuLan and MusicFM-MuLan under the same contrastive learning setup, suggesting stronger initialization.
  • Layer-wise analysis: Semantic tasks generally perform best at higher MuQ layers, while acoustic tasks generally perform best at lower layers.The analysis associates higher layers with semantic information and lower layers with low-level audio characteristics.

Discussion

The discussion identifies limitations of Mel-RVQ and MuQ’s iterative-training design while outlining future work on codebook count, layer selection, model sizes, and virtual classes. Mel-RVQ remains fast to train but requires music-data pre-training, unlike the out-of-the-box BEST-RQ approach.

  • Additional explanation of Mel-RVQ: The eight-codebook setting does not establish that additional codebooks are meaningless, motivating study of their effects and limits.The authors suggest more codebooks could clarify why Mel-RVQ works and where its advantage ends.
  • Additional explanation of Mel-RVQ: Layer 10 in iterative training was selected empirically rather than through rigorous comparison, and other layers were not explored because of limited computational resources.The authors consider other-layer iteration a potential source of better results.
  • Potential role of virtual classes in SSL: Unused codes in BEST-RQ’s low-utilization random projection quantizer may serve a role similar to virtual classes.The discussion connects this possibility to the virtual-class concept introduced in prior work.
  • Limitation and future work: Mel-RVQ requires music-data pre-training before SSL targeting, although its training takes less than 1 hour.This requirement makes the approach less out-of-the-box than BEST-RQ.
  • Limitation and future work: Future work will explore different model sizes, including a 90M base size, for varied physical environments and computational conditions.The stated goal is to adapt MuQ to differing deployment and resource requirements.

Conclusion

The study introduces MuQ, a self-supervised music representation model using Mel-RVQ targets, and MuQ-MuLan, a joint music-text embedding model. MuQ improves over prior models including MERT and MusicFM on the MARBLE benchmark.

  • Conclusion: MuQ is a self-supervised music representation and understanding model that uses Mel-RVQ targets with a linear projection RVQ structure for Mel spectra.The authors describe this as a simple yet powerful self-supervised learning target.
  • Conclusion: MuQ-MuLan is a joint music-text embedding model presented alongside MuQ.
  • Conclusion: MuQ shows significant improvements over previous models including MERT and MusicFM on the MARBLE benchmark.

Ethic Discussion

The authors acknowledge ethical concerns around MuQ and describe safeguards for privacy, bias, and audio distribution. They also frame MuQ as a potentially beneficial tool for musicians and creators.

  • The authors state that data handling follows ethical guidelines, protects personal information, and seeks to keep MuQ free from biases.
  • They guarantee that no audio will be distributed or shared.
  • The authors believe MuQ can positively impact the community by advancing automated music analysis and support musicians and creators.

Appendix I: Evaluation Details · Downstream Tasks

This appendix details the evaluation’s nine downstream tasks, including their datasets and evaluation metrics. Most tasks use the implementation of the MARBLE benchmark.

  • Downstream Tasks: The evaluation covers 9 downstream tasks.The appendix provides a detailed description of these tasks.
  • Downstream Tasks: Each downstream task is described together with its respective dataset.The appendix identifies datasets used across the evaluation tasks.
  • Downstream Tasks: Each downstream task is also described with its evaluation metrics.The appendix reports the metrics associated with the tasks.
  • Downstream Tasks: Most downstream tasks are based on the MARBLE-benchmark implementation.The benchmark implementation is attributed to Yuan et al. 2023.
  • Appendix I: Evaluation Details: The appendix presents the downstream-task descriptions as part of the evaluation details.The passage frames the material as a detailed account of the evaluation.
  • Appendix I: Evaluation Details: The downstream-task evaluation combines task descriptions, datasets, and metrics.These three elements comprise the appendix’s stated coverage.

Genre classification … Pitch classification

The paper evaluates music representations across nine downstream tasks spanning genre, key, emotion, singer, vocal technique, tagging, structure, instrument, and pitch classification. These experiments use established datasets and task-specific protocols, labels, splits, and metrics.

  • Genre classification: Genre classification uses 10-second windows from GTZAN’s 10 genres and reports test accuracy on the standard fail-filtered split.GTZAN contains 100 30-second files per genre; window-level features are combined for downstream prediction.
  • Key detection: Key detection predicts 24 major/minor pitch-scale classes using Giantsteps for testing and Giantsteps-MTG-keys for training and validation.Performance is evaluated with refined accuracy using error tolerance.
  • Emotional analysis: Emotional analysis predicts valence and arousal scores on 744 deduplicated Emomusic songs and evaluates regression with R2A and R2V.The original dataset has 1000 45-second clips, and the split follows Castellon, Donahue, and Liang (2021).
  • Singer identification: Singer identification classifies 20 Vocalset singers with a 12:8:5 train-validation-test split, using accuracy as the evaluation metric.All three splits contain the full singer set.
  • Vocal technique detection: Vocal technique detection classifies 10 techniques from 3-second VocalSet clips, using 15 singers for training and validation and 5 for testing.Because clips are segmented into 3 seconds, the task predicts technique type without onset or offset detection.
  • Music tagging: Music tagging uses all 25,863 MagnaTagATune clips with 188 binary tags and evaluates macro-average ROC-AUC, multiclass AUC, and AP / PR-AUC over the top-50 tags.The clips cover diverse genres and come from 5,223 songs, 445 albums, and 230 artists.
  • Music structure analysis: Music structure analysis predicts frame-wise functional and boundary activation curves on Harmonix Set, with frame-wise functional-label accuracy as the metric.Function transitions are smoothed with 1-second ramps, while boundary curves use 0.6-second durations.
  • Instrument classification; Pitch classification: Instrument and pitch classification use NSynth-based data to identify instruments or classify short clips into 128 pitch categories, respectively.Instrument classification allows multiple instruments per segment, while pitch classification is evaluated with accuracy.
Loading 2501.01108v2…