PaperlayerView exact revision on arXiv

Source-linked AI summary

MuSViT: A Foundation Vision Model for Sheet Music Representation

Carlos Penarrubia, Antonio Rios-Vila, Eliseo Fuentes-Martinez, Juan C. Martinez-Sevilla, Francisco J. Castellanos, María Alfaro-Contreras, Jorge Calvo-Zaragoza

arXiv:2606.31811v1cs.CV

TL;DR

Sheet music lacks a domain-specific foundation vision model, limiting reusable representations for structured score understanding. MuSViT pre-trains a ViT with masked autoencoders on 9.7 million IMSLP pages and generally outperforms general-purpose encoders and task-specific methods across four downstream tasks, while aligning with symbolic musical content.

  • Problem

    Sheet music lacks a foundation model tailored to its structured symbolic visual language, while much digitized score material remains raw page images rather than searchable structured data.

  • Method

    MuSViT is a ViT encoder pre-trained with masked autoencoders on 9.7 million IMSLP score pages through a two-stage synthetic-to-real curriculum.

  • Results

    Across four downstream tasks, MuSViT generally outperforms other vision encoders under linear probing and task-specific state-of-the-art systems under fine-tuning.

  • Takeaways & Limitations

    MuSViT representations align more closely with symbolic musical content than general-purpose vision encoders, supporting their use as a foundation backbone for sheet music understanding.

  • Takeaways & Limitations

    Pre-training data consists of publicly available scans that are public domain or permissively licensed under IMSLP terms.

Abstract

from arXiv · show

Foundation models have transformed vision and language processing by providing rich, reusable representations that transfer across diverse tasks. Sheet music, as a visual encoding of musical language, lacks such a strong domain-specific backbone. We introduce MuSViT (Music Score Vision Transformer): the first foundation vision model for sheet music representation -- a ViT encoder pre-trained via Masked Autoencoders on 9.7 million pages from the IMSLP. To handle the complexity of real-world scores, we adopt a two-stage curriculum: a synthetic warm-up on typeset scores followed by large-scale training on the full IMSLP corpus. We evaluate MuSViT on four downstream tasks -- full-page and staff-level music score recognition, music symbol detection, and score difficulty classification -- under two scenarios: linear probing (frozen encoder) and fine-tuning. Under linear probing, MuSViT consistently outperforms modern vision encoders, revealing that general-purpose representations, regardless of scale, fall systematically short on the structured symbolic properties of musical notation. Under fine-tuning, MuSViT generally improves upon task-specific state-of-the-art methods. An additional embedding-transcription consistency analysis reveals that MuSViT encodes symbolic musical structure directly in its representation space -- unlike other encoders, whose embeddings do not correlate with music notation content. These results establish MuSViT as a foundation backbone for sheet music understanding.

1 Introduction

Sheet music remains largely inaccessible because most digitized scores are raw images lacking structured transcription, while task-specific OMR systems are brittle across notation styles and artifacts. MuSViT addresses this gap as a sheet-music foundation model whose domain-specific representations outperform general-purpose encoders and support diverse downstream tasks.

  • Motivation: Most digitized scores remain inaccessible because they are raw page images without structured formats for indexing, retrieval, or automated analysis.Manual transcription is costly, leaving much of this cultural material underutilized.
  • Motivation: Traditional OMR systems are brittle, with performance degrading sharply on unfamiliar notation styles, engraving conventions, and visual artifacts.These systems typically use task-specific end-to-end models or multi-stage pipelines trained on limited annotated datasets.
  • MuSViT: MUSVIT is the first vision foundation model for sheet music, pre-trained via Masked Image Modeling on 9.7 million IMSLP pages.It reconstructs masked regions of full-page sheet music images without annotated data, learning structural and symbolic notation properties.
  • Evaluation: MUSVIT is evaluated on four tasks under linear probing with a frozen encoder and fine-tuning: full-page recognition, staff-level recognition, symbol detection, and difficulty classification.These tasks span page-level transcription, staff-level transcription, dense object localization, and document-level estimation.
  • Results: Under linear probing, MUSVIT consistently outperforms general-purpose vision encoders, while fine-tuning surpasses task-specific state-of-the-art methods on three tasks and matches the best result on the fourth.Embedding-transcription consistency further shows closer alignment with symbolic musical content than general-purpose vision models.

2 MuSViT: Music Score Vision Transformer

MuSViT is a self-supervised ViT backbone for sheet-music images, pre-trained with MAE through a two-stage synthetic-to-real curriculum. Its design uses fine-grained, spatially aware representations and large-scale IMSLP data to capture the structured visual language of notation.

  • Model overview: MuSViT learns reusable visual representations for sheet music as a general backbone across diverse downstream tasks.It is a self-supervised Vision Transformer designed specifically for sheet-music images.
  • Pre-training strategy: MAE reconstructs heavily masked score regions, forcing the encoder to infer symbol identity, duration, and vertical placement rather than interpolate image textures.The high masking ratio can occlude entire measures and long symbol sequences.
  • Pre-training strategy: The two-stage curriculum begins with synthetic DeepScoresV2 warm-up and transfers to real-world IMSLP adaptation because direct IMSLP training produces unstable convergence.The real-world stage uses 70% masking, 1,024 × 1,024 full pages, P = 16, and 4,096 patches per image.
  • Pre-training strategy: An ablation shows that omitting synthetic warm-up causes dimensional collapse, with the decoder predicting average patches instead of music-notation structures.This confirms the necessity of the proposed curriculum for stable structured learning.
  • Architecture: The encoder is a 12-layer ViT with approximately 85M parameters, d = 768 embeddings, and 2D sinusoidal positional encodings that represent vertical position.A lightweight variant, MUSVITLight, uses d = 384 and approximately 25M parameters for limited-computation settings.
  • Training data: The pre-training corpus contains 9.7 million public-domain IMSLP pages from around 400,000 musical works, spanning diverse historical, engraving, and musical textures.Each page is rendered as an RGB image, covering material from monophonic vocal lines to dense orchestral scores.

3 Evaluation on Downstream Tasks

MuSViT is evaluated across four sheet-music understanding tasks using frozen-encoder linear probing and joint fine-tuning. It consistently provides strong music-specific representations, outperforming general-purpose encoders under linear probing and generally surpassing task-specific state of the art after adaptation.

  • Evaluation Setup: The evaluation spans full-page recognition, staff-level recognition, symbol detection, and score difficulty classification, covering transcription, spatial localization, and document-level semantics.These tasks probe progressively different levels of sheet-music understanding.
  • Evaluation Setup: Under linear probing, MuSViT remains frozen while lightweight task-specific heads isolate the intrinsic quality of its learned representations.Fine-tuning instead jointly trains the encoder and task head, allowing adaptation to each downstream objective.
  • Full-Page Recognition: 16.4% average SER makes MuSViT more than 2.5× better than the best general-purpose baseline, PaliGemma 2 (48.6%), and better than the state of the art (20.0%) for full-page recognition under linear probing.General-purpose encoders produce SER values in the 48–62% range.
  • Full-Page Recognition: 10.9% average SER under fine-tuning outperforms the full-page recognition state of the art by 9.1 points, with gains consistent across both corpora.The improvement is particularly large on Polish Digital Scores.
  • Symbol Detection: 79.7% mAP and 80.7% w-mAP are the best linear-probing symbol-detection results, while both MuSViT variants exceed 96% mAP50 after fine-tuning versus the state of the art (90.5%).The fine-tuned detector uses Faster R-CNN with parameter-efficient LoRA adaptation.
  • Difficulty Classification: 54.2% Acc0 and 89.3% Acc1 under fine-tuning surpass the difficulty-classification state of the art by 15.8 and 5.0 points, while linear probing already achieves 47.4% Acc0 and 87.1% Acc1.MuSViTLight closely matches the full model in both scenarios.

4 Embedding-Transcription Consistency Analysis

The analysis tests whether frozen visual embeddings preserve symbolic musical content by comparing embedding-space distances with transcription distances. MuSViT is evaluated against general-purpose encoders, whose embeddings show low negative correlations with musical content.

  • Analysis setup: The analysis measures how strongly visual embedding spaces align with symbolic musical content by comparing pairwise embedding distances with transcription distances.It uses Euclidean embedding distance alongside Levenshtein edit distance and histogram distance over token frequencies.
  • Analysis setup: Experiments use the Mozarteum and Polish Digital Scores datasets, extracting flattened token representations from frozen encoders and comparing MuSViT with four general-purpose encoders.The comparison includes PaliGemma 2, Kosmos-2.5, Qwen3-VL, and DINOv3-7B under identical frozen conditions.
  • Consistency metric: Pearson and Spearman correlations between embedding and transcription distances quantify whether transcriptionally similar images are placed closer in the embedding space.Higher positive correlations indicate stronger preservation of symbolic similarity.
  • Results: All general-purpose encoders yield low negative correlations across both transcription distance measures and both correlation metrics, indicating slight anti-correlation with musical content.The reported interpretation is that images with similar musical content may not be placed closer in these embedding spaces.

5 Conclusions

MuSViT is introduced as the first foundation vision model for sheet music representation, pretrained on 9.7 million IMSLP score pages through a synthetic-to-real curriculum. Its representations capture music-notation structure and generally outperform other vision encoders and task-specific state-of-the-art systems across four downstream tasks.

  • Contributions: MuSViT is the first foundation vision model for sheet music representation, using a ViT pretrained via MAE on 9.7 million IMSLP score pages.Training follows a two-stage curriculum progressing from synthetic scores to real-world data.
  • Representation: MuSViT representations are grounded in the structure of music notation.
  • Downstream evaluation: Across four downstream tasks, MuSViT generally outperforms other vision encoders under linear probing and task-specific state-of-the-art systems under fine-tuning.

A Supplementary Material

The supplementary material provides terminology, reconstruction examples, training details, ablations, architectural specifications, datasets, evaluation protocols, computational costs, and additional representation analyses.

  • It includes a terminology glossary for non-music readers and MuSViT reconstruction examples.
  • It reports pre-training hyperparameters, a two-stage curriculum ablation, and detailed architecture specifications for each MuSViT variant.
  • It provides additional IMSLP training-data examples and downstream-task datasets with representative examples.
  • It documents general-purpose encoder specifications, fine-tuning protocols, and per-dataset results for each downstream task.
  • It covers fine-tuning general-purpose models, computational cost, and supplementary representation analyses.

A.1 Terminology Glossary

Table 11 defines key music-notation and document-analysis terminology used throughout the paper, helping readers less familiar with the music domain.

  • A.1 Terminology Glossary: The definitions are intended to assist readers less familiar with the music domain.This reader-support purpose is stated explicitly in the surrounding text.
  • A.1 Terminology Glossary: Table 11 is a glossary of music notation and document analysis terminology.It provides definitions of key terms used throughout the paper.
  • A.1 Terminology Glossary: The glossary covers terminology from both music notation and document analysis.These are the two domains explicitly identified in the table’s description.

A.2 MuSViT Reconstruction Examples

MuSViT reconstructs detailed sheet-music notation despite 70% patch masking, including regions where entire measures are occluded. These examples support learning musical structure rather than only low-level texture statistics.

  • Reconstruction quality: MuSViT accurately recovers fine-grained notation, including noteheads, stems, accidentals, and staff lines.The qualitative examples demonstrate reconstruction of multiple visual elements in the original score regions.
  • Structural representation: The reconstructions support the hypothesis that pre-training learns genuine musical structure rather than low-level texture statistics.The evidence is qualitative and based on reconstruction examples under heavy masking.
  • Reconstruction setup: 70% of patches are removed from each masked input, including cases where entire measures are occluded.Each panel compares the masked input, MuSViT reconstruction, and original score region.

A.3 Pre-Training Details

MuSViT uses a two-stage masked-autoencoder pre-training curriculum, beginning with structured synthetic score crops and progressing to full IMSLP pages. The curriculum increases resolution, patch count, and masking ratio to expose the encoder to increasingly complex musical documents and long-range structure.

  • Stage 1: Synthetic Warm-Up: Stage 1 pre-training uses synthetic DeepScoresV2 crops at 512 × 512 resolution, with 1,024 patches per image and a 50% masking ratio.This stage serves as a structured warm-up before exposure to real-world data.
  • Stage 2: IMSLP Pre-Training: Stage 2 pre-training uses full IMSLP pages at 1024 × 1024 resolution, with 4,096 patches per image and a 70% masking ratio.This stage adapts the model to the full complexity of scanned music documents.
  • Curriculum Design: The higher masking ratio in Stage 2 forces the encoder to reason over long-range musical structure.Stage 2 therefore extends the structured warm-up toward the complexity of real-world score pages.

A.4 Two-Stage Curriculum Ablation

The ablation compares the two-stage curriculum with single-stage MAE training directly on IMSLP. Single-stage training causes early dimensional collapse, concentrating encoder variance in a few dimensions and leading the decoder to predict average patches rather than music-notation structures.

  • A.4 Two-Stage Curriculum Ablation: The ablation tests whether the two-stage curriculum is necessary by comparing it with single-stage MAE training directly on IMSLP.The comparison is presented in Fig. 6.
  • A.4 Two-Stage Curriculum Ablation: Single-stage training produces dimensional collapse despite reducing reconstruction loss.The encoder concentrates most variance in a few dimensions, and its effective rank drops early during training.
  • A.4 Two-Stage Curriculum Ablation: The decoder predicts average patches rather than music-notation structures under single-stage training.This behavior accompanies the encoder’s early effective-rank decline.

A.5 Architecture Details

MuSViT and MuSViTLight use the standard ViT design, differing primarily in embedding width. MuSViTLight reduces parameters while preserving the same patch size and layer count as MuSViT.

  • A.5 Architecture Details: Both MUSVIT variants follow the standard ViT design.Their architectural configurations are described as standard ViT designs.
  • A.5 Architecture Details: d = 768 versus d = 384 reduces the parameter count from approximately 85M to 25M.MUSVIT uses the larger embedding dimension, while MUSVITLight uses the smaller one.
  • A.5 Architecture Details: P = 16 and 12 layers are shared by both models, maintaining comparable depth while reducing width.The variants therefore differ in width rather than patch size or layer count.

A.6 IMSLP Training Data … A.11.2 Nearest-Neighbor Analysis

The appendix documents MuSViT’s diverse IMSLP pre-training corpus, downstream evaluation protocols, and task-specific results across recognition, detection, and difficulty classification. It further shows that MuSViT’s representations align with musical notation spatially and transcriptionally, unlike general-purpose encoders.

  • A.6 IMSLP Training Data / A.7 Downstream Task Datasets: The IMSLP corpus spans historical periods, mensural and CWMN notation, handwritten and typeset engraving, and monophonic and polyphonic textures.Downstream datasets cover full-page recognition, staff-level recognition, DeepScoresV2 symbol detection, and score difficulty classification.
  • A.8 General-Purpose Encoder Specifications / A.9 Downstream Tasks / A.9.1 Full-Page Music Score Recognition: Exact Hugging Face checkpoints are specified for all general-purpose encoder baselines, while downstream sections report task heads, linear probing, fine-tuning, and per-dataset results.Full-page recognition uses an autoregressive Transformer decoder and measures normalized edit-distance Symbol Error Rate (SER).
  • A.9.1 Full-Page Music Score Recognition: 17.6% on Mozarteum and 15.2% on Polish Digital Scores under linear probing, while all general-purpose baselines exceed 46% on both corpora.The 15.2% Polish Digital Scores result also beats the task-specific state of the art at 25.8%.
  • A.9.2 Staff-Level Music Score Recognition: Under linear probing, MuSViT achieves the best SER on four of five staff-level corpora—Capitan 26.2%, Guatemala 7.7%, FMT 29.7%, and Il Lauro Secco 7.2%—while Qwen3-VL leads AMDC at 17.4% versus 21.0%.Fine-tuning uses rank-8 LoRA adaptation; MuSViT variants surpass the state of the art on Guatemala and AMDC, reaching 1.6% and 15.0% with MuSViTLight.
  • A.9.4 Score Difficulty Classification: Fine-tuning enables MuSViT and MuSViTLight to outperform the state of the art on all three difficulty-classification corpora and both metrics, including FreeScores Acc0 at 61.9% versus 47.3%.Under linear probing, results vary by corpus: MuSViTLight leads Can I Play It? Acc0 at 31.2% and MuSViT leads PianoStreet Acc0 at 54.3%, while PaliGemma 2 leads FreeScores Acc0 at 62.9%.
  • A.10 Fine-Tuning General-Purpose Models: Fine-tuning improves general-purpose encoders, but MuSViT remains superior under the same downstream protocol, with 85M parameters and 106 GFLOPs per image and approximately 0.3M parameters updated per task through rank-8 LoRA.Compared with general-purpose encoders, MuSViT is 5–82× smaller and 16–260× more efficient.
  • A.11 Sheet Music Representation Analyses / A.11.1 Attention Heat Map Analysis / A.11.2 Nearest-Neighbor Analysis: PCA heat maps show MuSViT concentrating activation on notation rows and symbols while suppressing margins and blank areas; nearest-neighbor curves likewise separate transcriptionally similar from dissimilar images, unlike general-purpose encoders.MuSViT and MuSViTLight maintain a clear gap across k ∈ {1, . . . , 25}, whereas general-purpose encoders show little or no separation.
Loading 2606.31811v1…