Source-linked AI summary
Instrument Classification of Solo Sheet Music Images
Kevin Ji, Daniel Yang, TJ Tsai
TL;DR
This paper asks whether instruments can be classified from sheet music images, complementing prior audio-focused recognition work. It converts scores into bootleg-score token sequences for text classification and reports improved accuracy with language-model pretraining and data augmentation.
Problem
The paper asks whether eight primarily monophonic instruments can be distinguished from sheet music images rather than audio.
Source: §1 · p. 1Method
It converts sheet music into bootleg-score word or subword sequences, pretrains language models on unlabeled data, and fine-tunes classifiers on labeled fragments.
Source: §1 · p. 1, §2.1 · p. 2, §2.2 · p. 2Results
RoBERTa instrument-classification accuracy rises from 34.5% to 42.9% with pretraining.
Source: §1 · p. 1Takeaways & Limitations
The results show that language-model pretraining and bootleg-score augmentation can improve classification of solo sheet music in this eight-instrument setting.
Source: §1 · p. 1, §6 · p. 4The method predicts an instrument for a single page of solo music; extending it to multiple-instrument sheet music remains future work.
Source: §6 · p. 4
Abstract
from arXiv · showhide
This paper studies instrument classification of solo sheet music. Whereas previous work has focused on instrument recognition in audio data, we instead approach the instrument classification problem using raw sheet music images. Our approach first converts the sheet music image into a sequence of musical "words" based on the bootleg score representation, and then treats the problem as a text classification task. We show that it is possible to significantly improve classifier performance by training a language model on unlabeled data, initializing a classifier with the pretrained language model weights, and then finetuning the classifier on labeled data. In this work, we train AWD-LSTM, GPT-2, and RoBERTa models on solo sheet music images from IMSLP for eight different instruments. We find that GPT-2 and RoBERTa slightly outperform AWD-LSTM, and that pretraining increases classification accuracy for RoBERTa from 34.5% to 42.9%. Furthermore, we propose two data augmentation methods that increase classification accuracy for RoBERTa by an additional 15%.
1. INTRODUCTION
The paper asks whether instruments can be distinguished from raw sheet music images, a task that could support segmentation of multi-instrument documents. It represents sheet music as text-like sequences and applies language-model pretraining to classification of eight primarily monophonic instruments.
1. INTRODUCTION
- The task is to classify eight primarily monophonic instruments from raw sheet music images, with potential use in segmenting documents containing multiple instrument parts.The instruments are violin, flute, clarinet, oboe, trumpet, cello, viola, and guitar.Source: §1 · p. 1
- Instrument characteristics in notation include range, monophony versus polyphony, and characteristic musical motions, beyond acoustic timbre.The paper studies how far these note characteristics distinguish the eight instruments.Source: §1 · p. 1
- The approach converts sheet music into “words” for text classification, focusing on primarily monophonic instruments where relatively little information lies in the vertical component.This contrasts with piano-composer classification, where much information is vertical.Source: §1 · p. 1
- The paper trains AWD-LSTM, RoBERTa, and GPT-2 on IMSLP sheet music and reports that RoBERTa accuracy rises from 34.5% to 42.9% with pretraining.Its contributions include an eight-way IMSLP benchmark and bootleg-score training and test-time augmentation.Source: §1 · p. 1
2. SYSTEM DESCRIPTION
The system converts sheet music into bootleg-score tokens, pretrains a language model, and fine-tunes a classifier on labeled fragments. At inference, it classifies a page either as one sequence or by averaging predictions across overlapping fragments.
2.1. Language Model Pretraining
- Each image becomes a 31 × N binary bootleg score encoding filled-notehead positions relative to staff lines, then columns are tokenized as words or subwords.AWD-LSTM uses column words; RoBERTa and GPT-2 use byte-pair-encoded subwords.Source: §2.1 · p. 2
- AWD-LSTM, GPT-2, and RoBERTa are pretrained on unlabeled sequences using next-token prediction or masked-token prediction.AWD-LSTM and GPT-2 predict the next token; RoBERTa predicts the identity of randomly masked tokens.Source: §2.1 · p. 2
2.2. Classifier Finetuning
- For classifier finetuning, fixed-length labeled fragments augment training data and enable class balancing, but make the classifier learn fragment rather than full-page classification.The classifier head is initialized with pretrained language-model weights and then finetuned on proxy-task fragments.Source: §2.2 · p. 2
2.3. Inference
- At inference, the proxy classifier processes a page as one variable-length sequence or predicts on overlapping fixed-length fragments whose outputs are averaged.The fragment-based method uses 50% overlap.Source: §2.3 · p. 2
3. EXPERIMENTAL SETUP
The experiments use IMSLP scores to assemble unlabeled and manually verified labeled datasets for eight instruments. The labeled data is split by piece, and fixed-length fragments form the proxy classification datasets.
3. EXPERIMENTAL SETUP
- IMSLP pieces were filtered by instrument tags, public-domain or Creative Commons status, and exclusion of manuscripts; trumpet and oboe selections also included specified ensemble tags.The filtered collection is called unlabeled because PDFs may contain accompaniment parts or filler pages; the most-downloaded PDF was selected when multiple PDFs existed.Source: §3 · pp. 2–3
- Table 1 reports piece and page counts for each instrument in the labeled and unlabeled datasets.The supplied caption identifies the table’s scope but gives no counts.Source: Table 1 · p. 3
- Manual review retained solo-instrument pages and discarded accompaniment, filler pages, corrupted PDFs, and mislabeled PDFs, yielding 7,627 valid pages.The classification split was by piece: 60% train, 20% validation, and 20% test; language-model training used a 90%/10% piece split.Source: §3 · p. 3
- The proxy datasets contain 3,600 training, 1,200 validation, and 1,200 test fragments per instrument, with fragment sizes of 64, 128, and 256.Figure 2 presents proxy-task results by pretraining condition and fragment size.Source: §3 · p. 3, Figure 2 · p. 3
4. RESULTS
The results compare four classifier architectures under different pretraining and fragment-size conditions. Pretraining improves performance, although the full-page RoBERTa macro F1 declines as fragment size increases.
4. RESULTS
- The evaluation compares CNN, AWD-LSTM, GPT-2, and RoBERTa; the CNN adapts a piano-roll feature-statistics classifier to this setting.The cited prior CNN approach assumes access to full symbolic music information.Source: §4 · p. 3
- RoBERTa macro F1 falls from 0.612 to 0.579 to 0.551 as fragment size rises from 64 to 128 to 256, while pretraining helps in all cases.The full-page task uses macro F1 because its data are imbalanced.Source: Figure 3 · p. 3, §4 · pp. 3–4
- GPT-2 accuracy on 64-word fragments rises from 32.1% without pretraining to 42.3% with labeled and 43.2% with unlabeled pretraining.Unlabeled pretraining includes language-model training on unlabeled data, followed by fine-tuning on labeled data and then the proxy task.Source: §4 · p. 3
- Averaging predictions over multiple fixed-length sequences yields better results for every model except CNN.The figure displays the best-performing inference type for each model.Source: §4 · p. 4
5. ANALYSIS
The analysis examines RoBERTa’s example predictions, confusion pairs, and data augmentation. Training augmentation improves results through K = 3, while test-time augmentation helps only in moderation.
5. ANALYSIS
- RoBERTa correctly identifies one guitar fragment and a flute fragment but misclassifies another guitar fragment as cello; monophonic inputs are generally more error-prone.The flute prediction recognizes the instrument’s frequent high notes above the staff.Source: §5 · p. 4
- The most common actual-predicted confusions are viola-flute (260), trumpet-oboe (259), and oboe-clarinet (230); oboe-to-clarinet errors outnumber the reverse (117).The authors attribute this asymmetry to clarinet’s wider range, which lets notes outside the oboe range rule out oboe but not clarinet.Source: §5 · p. 4
- With K = 3 and L = 2, RoBERTa accuracy rises from 42.9% to 58.8% on the proxy task, and full-page macro F1 rises from .611 to .691.More training augmentation helps through K = 3; test-time augmentation improves proxy accuracy through L = 2 but degrades performance beyond that, possibly by losing instrument-range information.Source: Figure 5 · p. 4, §5 · p. 4
- Training augmentation shifts bootleg scores by up to K staff-line positions, while test-time augmentation averages predictions across shifts up to L positions.Training shifts correspond to key transpositions; K = 3 generates seven times more data.Source: §5 · p. 4
6. CONCLUSION
The paper proposes classifying a single page of solo instrumental sheet music by converting it into words and applying a text classifier. It reports benefits from language-model pretraining and two data-augmentation methods.
6. CONCLUSION
- The proposed method predicts the instrument on a single page of solo instrumental sheet music, using a sequence of words as input to a text classifier.The conclusion reports that language-model pretraining improves performance and that two data-augmentation types further improve classification.Source: §6 · p. 4