Source-linked AI summary
Codified audio language modeling learns useful representations for music information retrieval
Rodrigo Castellon, Chris Donahue, Percy Liang
TL;DR
Labeled data for many MIR tasks are scarce, and the use of codified audio language modeling as MIR pre-training had not yet been explored. This paper probes Jukebox's learned audio representations with shallow models and finds 30% higher average performance than the best tagging-pre-trained representation across four tasks.
Problem
Method
Results
Takeaways & Limitations
The authors conclude that modeling audio instead of labels learns richer representations for MIR, and identify CALM as a promising pre-training strategy.
Source: §6 · p. 6, §8 · p. 7Jukebox uses genre and artist labels as conditioning information, so transfer from this model should not be considered unsupervised.
Source: §2 · p. 2
Abstract
from arXiv · showhide
We demonstrate that language models pre-trained on codified (discretely-encoded) music audio learn representations that are useful for downstream MIR tasks. Specifically, we explore representations from Jukebox (Dhariwal et al. 2020): a music generation system containing a language model trained on codified audio from 1M songs. To determine if Jukebox's representations contain useful information for MIR, we use them as input features to train shallow models on several MIR tasks. Relative to representations from conventional MIR models which are pre-trained on tagging, we find that using representations from Jukebox as input features yields 30% stronger performance on average across four MIR tasks: tagging, genre classification, emotion recognition, and key detection. For key detection, we observe that representations from Jukebox are considerably stronger than those from models pre-trained on tagging, suggesting that pre-training via codified audio language modeling may address blind spots in conventional approaches. We interpret the strength of Jukebox's representations as evidence that modeling audio instead of tags provides richer representations for MIR.
1. INTRODUCTION
The paper asks whether modeling music audio can provide richer MIR representations than pre-training on labels, a question made relevant by scarce labels for downstream tasks. It repurposes Jukebox representations for shallow probes and reports stronger average performance than tagging-pre-trained representations across four tasks.
1. INTRODUCTION
- Conventional MIR transfer commonly reuses representations pre-trained on labeled tasks, but labeled data for many important downstream tasks are scarce.The authors propose that directly modeling music audio, rather than labels, could yield richer representations.Source: §1 · p. 1
- Prior contrastive audio pre-training had not exceeded label-based pre-trained models on downstream tasks, while Jukebox had demonstrated music generation from codified audio.The paper extends codified audio language modeling [17,1] from generation to discriminative MIR tasks.Source: §1 · p. 1
- 30% stronger average performance than the best tagging-pre-trained representation was achieved by Jukebox features across tagging, genre classification, key detection, and emotion recognition.The paper also reports that the probing setup is competitive with purpose-built state-of-the-art methods on several tasks.Source: §1 · p. 1
- Jukebox features are extracted from its learned language model and used as inputs to shallow probing models on task-specific labeled datasets.The paper describes Jukebox as a Transformer language model trained on codified audio from 1M songs.Source: §1 · p. 1, Figure 1 · p. 2
2. CALM PRE-TRAINING
The section motivates codified audio language modeling as a possible source of representations for discriminative MIR, which had not yet been explored. It describes Jukebox's audio-code language model and notes that genre and artist conditioning mean its transfer should not be called unsupervised.
2. CALM PRE-TRAINING
- CALM codifies continuous audio into lower-rate discrete codes, trains a language model on those codes and optional metadata, then decodes generated sequences to audio.The decoding stage is not necessary for transfer learning.Source: §2 · p. 2
- CALM had not yet been explored as a pre-training strategy for discriminative MIR, despite promising music-generation results and the hypothesis that generative representations could help discriminative tasks.The authors also draw motivation from NLP work on generative and self-supervised pre-training.Source: §2 · p. 2
- Jukebox was trained on codified audio from 1.2M songs, with genre and artist labels supplied as conditioning information.Although CALM generally does not require labels, transfer from Jukebox should not be considered unsupervised.Source: §2 · p. 2
- Jukebox uses a 2M-parameter VQ-VAE to encode 44.1-kHz audio as approximately 345-Hz discrete codes and a 5B-parameter Transformer decoder to model them.The code vocabulary has 2048 entries; the paper reports that codified-audio language models learn longer-term music structure while using less memory than models of raw audio.Source: §2 · p. 2
3. EXTRACTING SUITABLE REPRESENTATIONS FROM JUKEBOX
The authors extract layer-wise activations from Jukebox as audio representations, reduce their size, and select a layer for probing. Their layer comparison finds the strongest average validation performance in the model's middle layers, with the middle-layer strategy reported as equally effective and more practical than alternatives.
3.1 Layer selection
- Across four downstream MIR tasks, normalized validation performance of linear probes is strongest on average for representations from the middle of Jukebox.The authors use layer 36 for subsequent probing.Source: Figure 2 · p. 3, §3.1 · p. 3
3. EXTRACTING SUITABLE REPRESENTATIONS FROM JUKEBOX
- The 5B-parameter, 72-layer Jukebox model yields 4800-dimensional activations per code position; extraction resamples and normalizes audio, codifies it, and feeds the codes to the language model.The model is used without lyrics conditioning, with placeholder genre and artist values, one-minute song length, and zero clip offset.Source: §3 · p. 3
- Mean pooling across time reduces more than 10GB of activations per 24-second clip to about 1MB, after which selecting the middle layer yields 4800 features per clip.The middle layer is used to reduce the computational cost of training shallow classifiers.Source: §3 · p. 3, §3.1 · p. 3
- The benchmark covers four tasks and reports each task's dataset size, relevant metrics, and required number of model outputs.The figure and table passages identify the scope of the layer comparison and task summary, respectively.Source: Table 1 · p. 3
3.1 Layer selection
- Using only the middle layer was equally effective and more computationally practical than both subsampling layers across the network and selecting relevant layers in a task-specific fashion.Source: §3.1 · p. 3
4. DOWNSTREAM TASK DESCRIPTIONS
The benchmark covers four downstream MIR tasks chosen to span dataset sizes and subjectivity. It uses established datasets and task-specific evaluation protocols.
4. DOWNSTREAM TASK DESCRIPTIONS
- The benchmark spans tagging, genre classification, key detection, and emotion recognition, covering varied dataset sizes and levels of subjectivity.Emotion recognition has 744 examples, compared with 26k for tagging; emotion is more subjective than key detection.Source: §4 · p. 3
4.1 Tagging
- Tagging is evaluated on MTT rather than MSD to avoid giving methods pre-trained on MSD an unfair benchmarking advantage.MTT and MSD each use 50 tags for 30-second clips, but their label distributions differ: MSD skews toward genre and MTT toward instrumentation.Source: §4.1 · p. 4
4.2 Genre classification
- Genre classification uses accuracy on GTZAN’s fault-filtered split, with seven of its ten genres also appearing in MSD’s tag list.The split addresses reported dataset issues.Source: §4.2 · p. 4
4.3 Key detection
- Key detection is a 24-way major/minor classification task, trained on Giantsteps-MTG and tested on Giantsteps using a weighted score.The clips are two-minute electronic dance music tracks; the score gives partial credit for reasonable errors such as predicting the relative minor.Source: §4.3 · p. 4
4.4 Emotion recognition
- Emotion recognition predicts valence and arousal from 744 45-second Emomusic clips, using static annotations and coefficient of determination scores.The authors created an artist-stratified split because the dataset has no standard split; reported prior-work numbers use different splits and are not directly comparable.Source: §4.4 · p. 4
5. PROBING EXPERIMENTS
The experiments compare shallow probes trained on several audio representations across four MIR tasks. The protocol accommodates differences among representations and evaluates selected models on task-specific test sets.
5.2 Probing protocol
- Linear models and one-layer MLPs are trained as probes, with grid search and validation-based early stopping selecting a model for each representation and task.The protocol uses task-specific objectives and stopping metrics, then evaluates the selected model on the task-specific test set.Source: §5 · p. 4, §5.2 · pp. 5–6
5.1 Descriptions of representations
- The comparison includes hand-crafted CHROMA and MFCC features, tagging-pre-trained CHOI and MUSICNN, and contrastively pre-trained CLMR alongside JUKEBOX.The available CLMR model was trained on far less audio than CHOI, MUSICNN, and JUKEBOX.Source: §5.1 · pp. 4–5
5. PROBING EXPERIMENTS
- JUKEBOX probes outperform probes on other pre-trained representations across all six metrics and are competitive with task-specific state of the art except on key detection.The table reports results across tagging, genre classification, key detection, and emotion recognition, plus an average across tasks.Source: Table 3 · p. 5
5.1 Descriptions of representations
- Representations are mean-pooled over time; selected layers and pooling choices differ across models, while JUKEBOX uses its middle layer.The JUKEBOX layer choice yields 4,800 features, similar to MUSICNN’s 4,194, mitigating a potential dimensionality advantage.Source: §5.1 · p. 5
- For short-clip tasks, CHOI and JUKEBOX use the first window; for key detection, all methods use 30-second windows whose predictions are ensembled at test time.The short clips are 30–45 seconds, while key-detection clips are two minutes.Source: §5.1 · p. 5
6. RESULTS AND DISCUSSION
JUKEBOX representations outperform other pre-trained representations on the reported downstream MIR comparisons. They show a pronounced advantage on key detection, while comparisons with task-specific state of the art remain qualified by the study’s scope.
6. RESULTS AND DISCUSSION
- 69.9 average performance for JUKEBOX is 30% higher than the best tagging-pre-trained representation, MUSICNN at 53.7.JUKEBOX also exceeds every other representation on each individual metric; these holistic comparisons vary in architecture, model size, and pre-training data.Source: §6 · p. 6
- JUKEBOX has substantially more key-detection information than other representations; all except CHROMA perform near the 15.0 majority-class score.The authors interpret this relative strength as evidence that CALM may address blind spots in other MIR pre-training paradigms.Source: §6 · p. 6
- JUKEBOX probing is competitive with task-specific state of the art on all tasks except key detection and averages only 4% below it.The comparison includes purpose-built methods using pre-training or trained from scratch; the cited supervised comparator uses a proprietary 10M-song dataset.Source: §6 · p. 6
- The authors identify possible routes for improvement, including scaling model and data, fine-tuning, and bidirectional masked-language-model pre-training.They present these as anticipated or speculative avenues rather than demonstrated improvements.Source: §6 · p. 6
7. RELATED WORK
Prior MIR transfer learning has relied on labels, metadata, and contrastive learning, while codified-audio language modeling has been explored for generation and other audio domains. This work situates its approach among those strands.
RELATED WORK
- MIR transfer learning has included unlabeled-audio features, shared audio-metadata embeddings, and using tagging-model outputs for music similarity.The cited examples include,, and, respectively.Source: §7 · p. 6
- Tagging-based pre-training on MSD became a predominant MIR strategy, later extended through convolutional architectures and other model designs.The paper uses models from Choi et al. and a more recent architecture as baselines.Source: §7 · p. 6
- Metadata pre-training offers cheaper labels with performance comparable to tagging; combining metadata with tags or co-listening data has also produced strong results [5] [7].The cited co-listening work uses proprietary data from 10M songs.Source: §7 · p. 6
- Contrastive learning has been proposed for MIR pre-training [55] [14], and this study compares with a model from Spijkervet and Burgoyne 2021 [14].Source: §7 · p. 7
- Codified-audio language modeling has been used for speech and urban-sound tasks and music generation, while raw-waveform language models tend to capture long-term dependencies less effectively [20] [57] [58] [59] [1] [21].Language modeling has also been applied to symbolic music and score-based transfer learning.Source: §7 · p. 7
8. CONCLUSION
The paper concludes that codified audio language modeling (CALM) is a promising pre-training strategy for MIR, learning richer representations by modeling audio rather than labels. It also points to repurposed NLP methods and large models trained on music catalogs as directions for MIR research.
8. CONCLUSION
- CALM is a promising MIR pre-training strategy because it learns richer representations by modeling audio instead of labels.Source: §8 · p. 7
- CALM lets MIR researchers repurpose NLP methodology, following the leverage that computer-vision methods have historically provided to MIR.Source: §8 · p. 7
- The authors identify enormous models pre-trained on large music catalogs as a direction that may break new ground on MIR tasks.Source: §8 · p. 7