Source-linked AI summary
MusicBERT: Symbolic Music Understanding with Large-Scale Pre-Training
Mingliang Zeng, Xu Tan, Rui Wang, Zeqian Ju, Tao Qin, Tie-Yan Liu
TL;DR
Symbolic music understanding is constrained by limited training data and by the structural and diverse information that makes NLP pre-training techniques difficult to transfer directly. MusicBERT addresses these issues with a large-scale corpus, OctupleMIDI encoding, and bar-level masking, achieving state-of-the-art performance across four evaluated tasks. Ablation studies verify the effectiveness of its designs.
Problem
Limited labeled and unlabeled symbolic-music data, plus music’s structural and diverse attributes, constrain representation learning and make direct NLP pre-training challenging.
Method
MusicBERT pre-trains on the Million MIDI Dataset using OctupleMIDI encoding and a bar-level masking strategy tailored to symbolic music.
Results
MusicBERT achieves state-of-the-art performance on melody completion, accompaniment suggestion, genre classification, and style classification, with analyses verifying its design components.
Takeaways & Limitations
Specialized encoding, masking, and large-scale symbolic-music data provide the basis for MusicBERT’s reported gains in music understanding.
Abstract
from arXiv · showhide
Symbolic music understanding, which refers to the understanding of music from the symbolic data (e.g., MIDI format, but not audio), covers many music applications such as genre classification, emotion classification, and music pieces matching. While good music representations are beneficial for these applications, the lack of training data hinders representation learning. Inspired by the success of pre-training models in natural language processing, in this paper, we develop MusicBERT, a large-scale pre-trained model for music understanding. To this end, we construct a large-scale symbolic music corpus that contains more than 1 million music songs. Since symbolic music contains more structural (e.g., bar, position) and diverse information (e.g., tempo, instrument, and pitch), simply adopting the pre-training techniques from NLP to symbolic music only brings marginal gains. Therefore, we design several mechanisms, including OctupleMIDI encoding and bar-level masking strategy, to enhance pre-training with symbolic music data. Experiments demonstrate the advantages of MusicBERT on four music understanding tasks, including melody completion, accompaniment suggestion, genre classification, and style classification. Ablation studies also verify the effectiveness of our designs of OctupleMIDI encoding and bar-level masking strategy in MusicBERT.
1 Introduction
MusicBERT addresses limited symbolic-music data and the mismatch between NLP pre-training and music structure through specialized encoding, masking, and a large corpus. It achieves state-of-the-art results across four music-understanding tasks.
- Existing embedding-based approaches have limited capability to learn powerful music representations because they use shallow structures and limited unlabeled data.
- Directly applying NLP pre-training to symbolic music is challenging because music contains structural and diverse attributes requiring specialized encoding and masking.Relevant attributes include bars, positions, tempo, instruments, and pitch.
- MusicBERT is a large-scale pre-trained model designed with music-specific encoding and masking strategies for symbolic music understanding.
- OctupleMIDI encodes each note as an eight-element tuple and produces shorter, more universal music sequences.The elements represent time signature, tempo, bar, position, instrument, pitch, duration, and velocity.
- Bar-level masking masks all tokens of the same type within a bar to avoid information leakage and encourage effective representation learning.This addresses the ease of predicting masked attributes from adjacent tokens when attributes repeat across a segment.
- More than 1 million songs form the Million MIDI Dataset, supporting pre-training, while MusicBERT achieves state-of-the-art results on four downstream tasks.The tasks are melody completion, accompaniment suggestion, genre classification, and style classification.
2 Related Works
Prior symbolic-music methods use pianoroll or MIDI-like encodings and shallow embedding approaches, but their representations retain inefficiencies. MusicBERT introduces OctupleMIDI to compress note information into shorter, more universal sequences.
- Earlier symbolic-music work learned embeddings by predicting music symbols from neighboring symbols, adapting word2vec-style approaches from NLP.
- Pianoroll encodings represent pitches and time as a binary matrix, but divide notes into multiple fixed intervals, especially inefficiently for long notes.
- MIDI-based approaches encode pieces with events such as note-on, note-off, and time-shift, while REMI adds note duration, bar, position, chord, and tempo.
- Even compressed MIDI-like methods require multiple tokens for one note’s attributes, position, and metadata, leaving further compression possible.
- OctupleMIDI reduces sequence length and supports various music genres by encoding each note’s information in one structured tuple.
- NLP pre-training commonly masks random tokens or contiguous spans, whereas symbolic music requires masking strategies that account for musical token dependencies.
3 Methodology
MusicBERT pre-trains a Transformer encoder on symbolic music using OctupleMIDI representations and music-specific masking. The methodology also includes a large corpus of 1.5 million deduplicated songs for pre-training.
- 3.1 Model Overview: MusicBERT pre-trains a Transformer encoder with masked language modeling, predicting masked tokens in input music sequences.
- 3.2 OctupleMIDI Encoding: OctupleMIDI represents each note as an 8-element token containing time signature, tempo, bar, position, instrument, pitch, duration, and velocity.
- 3.2 OctupleMIDI Encoding: The encoding uses hierarchical bar and position values to represent note onset times, with bar-level timing and finer within-bar position granularity.
- 3.3 Masking Strategy: Bar-level masking masks same-type elements within a bar simultaneously, addressing information leakage caused by regularities among adjacent musical tokens.
- 3.4 Pre-training Corpus: The Million MIDI Dataset contains 1.5 million songs and 2 billion octuple tokens after cleaning and deduplication.
4 Experiments and Results
MusicBERT is evaluated through pre-training, four downstream music-understanding tasks, and ablations of its encoding, masking, and pre-training components. It achieves strong results across these evaluations, with method analyses supporting the proposed designs.
- Downstream evaluation: Melody completion retrieves the most matched phrase, while accompaniment suggestion retrieves the most related harmonic phrase using MAP and HITS@k metrics.
- Downstream evaluation: MusicBERT is fine-tuned on melody completion, accompaniment suggestion, genre classification, and style classification, achieving state-of-the-art results on all four tasks.
- Downstream evaluation: Genre and style classification are evaluated as multi-label tasks on TOP-MAGD and MASD using five-fold cross-validation and F1-micro.
- Method analysis: OctupleMIDI outperforms REMI and CP encodings, while reducing Transformer encoder complexity to 1/16 of REMI-like encoding and 1/4 of CP-like encoding on phrase-level tasks.
- Method analysis: Bar-level masking outperforms octuple masking and random masking at the same masking ratio, boosting downstream results.
- Method analysis: Pre-training produces much better scores than training without pre-training across the four downstream tasks, demonstrating its critical role in symbolic music understanding.
5 Conclusion
The paper concludes that MusicBERT addresses symbolic music pre-training with specialized encoding, masking, and corpus designs. It achieves state-of-the-art performance on four evaluated tasks, while future work targets additional music-understanding applications.
- MusicBERT combines OctupleMIDI encoding, bar-level masking, and a corpus containing more than 1 million music songs for symbolic music understanding.
- MusicBERT achieves state-of-the-art performance on melody completion, accompaniment suggestion, genre classification, and style classification.
- Future work will apply MusicBERT to chord recognition and structure analysis.