Source-linked AI summary
Learning Features of Music from Scratch
John Thickstun, Zaid Harchaoui, Sham Kakade
TL;DR
Classical-music machine learning lacked a large publicly available labeled dataset suitable for supervised learning. The paper introduces MusicNet and evaluates note prediction with spectrogram-based and end-to-end models. End-to-end models learn frequency-selective low-level filters and modestly outperform comparable spectrogram features, while the dataset remains skewed toward Beethoven and solo piano.
Problem
Classical-music research lacked a large publicly available labeled dataset for note prediction, and existing datasets contained less than 7 minutes of labeled music.
Method
The paper introduces MusicNet and evaluates spectrogram, neural-network, and convolutional-neural-network architectures on a multi-label note-prediction task.
Results
Learned features modestly outperform spectrogram features and form frequency-selective filters similar in spirit to spectrogram representations.
Takeaways & Limitations
End-to-end models trained for note prediction can learn frequency-selective low-level representations of musical audio from raw data.
Takeaways & Limitations
MusicNet is skewed toward Beethoven and solo piano, with instruments such as flute and oboe under-represented.
Abstract
from arXiv · showhide
This paper introduces a new large-scale music dataset, MusicNet, to serve as a source of supervision and evaluation of machine learning methods for music research. MusicNet consists of hundreds of freely-licensed classical music recordings by 10 composers, written for 11 instruments, together with instrument/note annotations resulting in over 1 million temporal labels on 34 hours of chamber music performances under various studio and microphone conditions. The paper defines a multi-label classification task to predict notes in musical recordings, along with an evaluation protocol, and benchmarks several machine learning architectures for this task: i) learning from spectrogram features; ii) end-to-end learning with a neural net; iii) end-to-end learning with a convolutional neural net. These experiments show that end-to-end models trained for note prediction learn frequency selective filters as a low-level representation of audio.
1 INTRODUCTION
MusicNet addresses the lack of large publicly available labeled classical-music data by providing a substantial corpus for supervised learning and evaluation. The paper uses note prediction to compare spectrogram-based and end-to-end architectures, whose learned features modestly outperform spectrogram features.
- Existing labeled classical-music datasets contained less than 7 minutes of music and were designed for evaluation rather than supervised training.
- Large labeled datasets had enabled end-to-end representation learning in other domains, motivating a concentrated effort to curate sharable music data.
- MusicNet provides 34 hours of human-verified aligned recordings with 1,299,329 labels as a publicly available learning resource.
- The paper formulates note prediction as a multi-label classification task and evaluates spectrogram, neural-network, and convolutional-neural-network architectures.
- Learned features modestly outperform spectrogram features and take the form of frequency-selective filters similar in spirit to spectrogram representations.
2 MUSICNET
MusicNet is a public collection of aligned labels for freely licensed classical recordings spanning varied instruments and chamber ensembles. Its label design supports polyphonic note prediction, while composer and instrument imbalances define important coverage boundaries.
- Related datasets differ in annotation scale, access, recording type, or instrument coverage, with some consisting entirely of piano music.
- MusicNet contains 330 freely licensed classical recordings averaging 6 minutes, with performances by varied instruments in small chamber ensembles and diverse studio and microphone conditions.
- The dataset includes labels for 513 distinct instrument/note combinations, with additional note-value combinations potentially expanding the class space by approximately an order of magnitude.
- MusicNet labels can overlap in time, producing polyphonic multi-labels.
- The dataset is skewed toward Beethoven and solo piano, leaving instruments such as flute and oboe under-represented.
3 DATASET CONSTRUCTION
The dataset construction procedure aligns freely licensed recordings with digital scores by comparing synthesized and recorded spectrogram features. Dynamic time warping supplies the alignment, using low-frequency spectral information selected for synthesis accuracy.
- MusicNet labels are obtained by aligning recordings with events from corresponding digital MIDI scores.
- The method maps the score into the performance space by synthesizing the score and comparing it with recorded spectrogram features.
- The alignment compares score and performance representations through a local cost function and maps them into a common feature space.
- Dynamic time warping solves the alignment problem exactly in O(mn) time and space under monotonic time-index constraints.
- The alignment uses log-spectrogram features with 2048-sample windows and 512-sample strides, yielding approximately 86 frames per second at 44.1kHz.
- The cost uses the ℓ2 norm on the lowest 50 spectrogram dimensions because synthesized performances reproduce high-frequency features less accurately.
4 METHODS
The paper frames note identification as multi-label classification and compares engineered spectrogram features with learned end-to-end representations. Learned models recover frequency-selective, spectrogram-like features, while dataset scale and window choices shape their quality.
- Note prediction: Note identification assigns each audio segment a 128-dimensional binary vector indicating which notes occur at its midpoint.A feature map feeds multivariate linear regression optimized with square loss.
- Learned filters: The log-ReLU network learns modulated-sinusoid weights that decay near segment boundaries, reflecting that center information is more useful for midpoint note labels.The segments are approximately 1/3 of a second long, and logarithmic compression improves performance over a standard ReLU network.
- Spectrogram analogues: A ReLUgram replaces the spectrogram’s x 7→x2 non-linearity with x 7→|x| and achieves similar classification performance.The construction follows from combining rectified responses to sinusoidal weight vectors.
- Data scale and regularization: Training on the full MusicNet dataset produces cleaner frequency-selective features than training on 65,000 monophonic examples, without regularization.Heavy regularization can clean small-data features but destroys classification performance; optimal reported performance is achieved without regularization.
- Model comparison: The study compares spectrogram features, end-to-end neural networks, and end-to-end convolutional neural networks for note prediction.The convolutional model uses shifted nearby segments and is controlled by receptive field, stride, and pooling regions.
5 RESULTS
The experiments evaluate models on a representative MusicNet test set using precision, recall, average precision, and precision-recall curves. Learned features modestly outperform spectrograms, with frequency distributions concentrating resolution in critical note regions.
- Evaluation setup: Three held-out recordings form a representative test set spanning most instruments and ensemble sizes in MusicNet.The recordings cover solo piano, a wind quintet, and a string quartet.
- Evaluation setup: Test segments are evenly spaced by 512 samples, and each is labeled with notes active at its midpoint.The segments span the first through 91st seconds of each recording, with substantial overlap for wider features.
- Evaluation metrics: Precision and recall vary with the note-prediction threshold, producing precision-recall curves; average precision is their area under the curve.The curves evaluate all test points and subsets with exactly one or exactly three labels.
- Model comparison: Learned features modestly outperform spectrogram features for comparable window sizes.The paper compares a linear 1024-point spectrogram model with MLP runs using 2048 raw samples.
- Model comparison: Network weights distribute frequencies similarly to MusicNet’s note distribution, giving higher resolution in the most critical frequency regions.Figure 5 compares note frequencies with learned-node frequencies in a 500-node, two-layer ReLU network.
A VALIDATING THE MUSICNET LABELS
The authors validate MusicNet’s score-performance alignments through listening tests and detailed musician analysis. The resulting estimates indicate a 4.0% average error rate, while several error types remain incompletely detected.
- Validation procedure: Alignment quality was assessed by mixing score-indicated sine tones into recordings and listening for dissonance caused by incorrect timing.Correct alignments make the tones overlay the performance; incorrect alignments sound dissonant.
- Validation procedure: Substantially incorrect alignments were rejected after listening to beginnings, random middle sections, and endings of recordings.Musical repeats were identified as the most common rejection cause.
- Validation results: 4.0% was the average alignment error rate across 30 sampled clips, including detailed analysis of 28 clips.Two clips were too richly structured and fast for precise analysis, although their alignments sounded substantially correct.
- Error analysis: The analysis does not capture every error type, especially some phrase-end releases and performance mistakes that preserve musical meter.Such meter-preserving mistakes were considered rare in professional recordings.
- Labeling boundary: Rolled chords receive a single starting time, so upper notes may be labeled before they occur in the performance.The authors state that this labeling may be desirable when labels supervise transcription.
- Error analysis: The observed alignment errors most commonly involved anticipations, delays, and missing ornaments or trills.Rare errors included performance mistakes and transcription errors in the score.
B ALIGNMENT PARAMETER ROBUSTNESS
Alignment quality depends on parameter tradeoffs and synthesis quality, while several feature choices have limited influence. Smaller strides improve label time resolution until system memory becomes limiting.
- Parameter selection: Audio featurization and alignment-cost parameters were selected through systematic exploration, with fine-tuning yielding only marginal gains.The choices were based on listening-test performance.
- Robustness findings: Alignment quality improves with synthesis quality, while decreasing stride improves label time resolution until limited by system memory.The precise phase-invariant feature specification has little effect on alignment quality.
- Parameter tradeoffs: Fourier window size trades time resolution against frequency resolution, and norm and spectrogram-cutoff choices balance competing outcomes.The ℓ1 norm can be overwhelmed by synthesis errors, whereas ℓ∞ ignores too much spectrogram signal.
C ADDITIONAL ERROR ANALYSIS
The paper reports additional evaluation using MIR-community accuracy and error measures. These results are presented in an MIREX-style format evaluated with the mir_eval library.
- Evaluation metrics: Models are evaluated on accuracy and error scores used by the music information retrieval community for Multi-F0 systems.Definitions and metric discussion are attributed to Poliner and Ellis (2007).
- Evaluation metrics: Table 4 reports MIREX-style statistics evaluated with the mir_eval library.
D PRECISION & RECALL CURVES
The section presents precision–recall curves for six model configurations, spanning spectrogram-based, multilayer perceptron, average-pooling, and convolutional architectures.
- The linear spectrogram model is evaluated through a precision–recall curve.
- Three raw-sample multilayer perceptrons are represented: 500 nodes with 2048 samples, 2500 nodes with 2048 samples, and 500 nodes with 16384 samples.
- The average pooling model is included among the evaluated architectures.
- The convolutional model is included among the evaluated architectures.
E ADDITIONAL RESULTS
The paper reports additional results on test-set splits described in Section 5, covering recordings of Mozart, Beethoven, and Bach.
- Additional results are reported on test-set splits described in Section 5.
- One split concerns the Soni Ventorum recording of Mozart’s Wind Quintet K375, MusicNet id 1819.
- A further split concerns the Kimiko Ishizaka recording of Bach’s Prelude in D major, MusicNet id 2303.