Source-linked AI summary
Full-Page Optical Music Recognition of Handwritten Monophonic Scores
Adrian Rosello, Antonio Ríos-Vila, David Rizo, Jorge Calvo-Zaragoza
TL;DR
Full-page OMR on handwritten music remains largely unexplored, despite its goal of transcribing entire pages without staff segmentation. This study evaluates handwritten monophonic collections using full-page pipelines and a generator for typeset and handwritten synthetic scores. SMT outperformed the baselines across three datasets, while typeset pretraining matched or surpassed handwritten pretraining.
Problem
The transfer of full-page OMR methods to handwritten monophonic music is uncertain, although full-page transcription requires both symbol recognition and layout understanding.
Source: §1 · pp. 1–2Method
The study compares full-page pipelines on handwritten monophonic collections and uses a generator to conduct controlled pretraining with typeset and handwritten scores.
Source: §1 · p. 2Results
SMT outperformed segmentation-based and unfolding baselines across three datasets with different layout conditions.
Source: §5 · p. 6Takeaways & Limitations
Typeset pretraining matched or surpassed handwritten pretraining on all datasets, pointing to layout and notation conventions rather than visual resemblance as the key synthetic-data factor.
Source: §5 · p. 6The formulation assumes a mapping from each sheet image to a token sequence, with likelihood computation dependent on the decoding strategy.
Source: §2 · p. 2
Abstract
from arXiv · showhide
Full-page end-to-end Optical Music Recognition seeks to transcribe entire music pages directly into symbolic notation, avoiding the limitations of traditional pipelines that rely on accurate staff segmentation. Recent Transformer-based architectures have achieved strong performance on typeset scores, relying on large-scale synthetic data for pretraining. However, their applicability to handwritten music remains largely unexplored. In this work, we study full-page transcription on handwritten monophonic collections and analyze the impact of synthetic pretraining in this setting. To investigate which factors are most relevant during pretraining, we introduce a generator capable of producing visually coherent full-page scores in both typeset and handwritten styles. Experiments on three real handwritten datasets provide a comparative evaluation of several full-page pipelines and different synthetic pretraining strategies. The results suggest that the benefits of synthetic pretraining are primarily associated with learning structural layout conventions rather than with visual similarity to the target handwriting.
1. Introduction
Full-page transcription must jointly recognize musical symbols and their page layout, but earlier approaches either separate staff analysis from transcription or were developed for narrower settings. This study examines transfer to handwritten monophonic music using controlled synthetic pretraining in typeset and handwritten styles.
1. Introduction
- Full-page OMR requires recognizing symbols and understanding how staves are arranged and relate across the page.The manual transcription it aims to replace is tedious and requires specialized notation knowledge.Source: §1 · p. 1
- Two-step systems risk propagating layout errors and lose inter-staff context by transcribing each staff in isolation.Staff-level CRNN systems nevertheless have shown strong recognition performance.Source: §1 · p. 1
- Unfolding was primarily evaluated on mensural notation, while SMT was designed for polyphonic typeset modern music.The paper asks how well these methods generalize beyond those settings.Source: §1 · p. 2
- Direct adjustment did not transfer successfully for Unfolding and SMT on handwritten monophonic sources.The figure reports 10-fold error rates after adjusting both models to the target handwritten corpora.Source: Figure 1 · p. 2, §1 · p. 2
- The study introduces a generator for typeset and handwritten full-page scores to test whether transfer difficulty reflects visual variability or monophonic structure.It presents the first study of full-page OMR on handwritten monophonic collections.Source: §1 · p. 2
2. Methodology
The methodology formulates full-page transcription as mapping a sheet image to a token sequence. It describes CTC greedy and autoregressive decoding strategies for producing that sequence.
2. Methodology
- Full-page transcription maps each sheet image to a sequence of tokens from a finite vocabulary.The model approximates this mapping by selecting probable tokens given an input image.Source: §2 · p. 2
- CTC greedy decoding selects framewise symbols, then applies a collapsing function that merges repeats and removes blanks.The resulting transcription contains at most K tokens from K framewise outputs.Source: §2 · pp. 2–3
- Autoregressive decoding generates tokens sequentially, conditioning each on the image and previously predicted tokens.Training uses teacher forcing, substituting the true prior-token prefix for the predicted prefix.Source: §2 · p. 3
3. Experimental Setup
The experimental setup covers evaluated pipelines, synthetic data generation, datasets, and evaluation. Its two-step comparison baseline segments pages into staves before CRNN transcription, with constraints from longer sequences.
3. Experimental Setup
- The experimental setup covers pipelines, synthetic data generation, datasets, and the evaluation protocol.Source: §3 · p. 3
3.1. Two-steps Pipeline
- CRNNf segments pages into staves with YOLOv11, concatenates them, then transcribes the sequence with a CRNN.The longer sequences challenge its bidirectional LSTM's limited long-range modeling, making this a fair but constrained full-page adaptation.Source: §3.1 · p. 3, Figure 2 · p. 3
3.2. Holistic Pipelines
The evaluated segmentation-free approaches are Unfolding and SMT. Unfolding is trained on target corpora, while SMT uses incremental-difficulty synthetic pretraining before real-data fine-tuning.
3.2. Holistic Pipelines
- The segmentation-free comparison includes target-trained Unfolding and SMT with curriculum-based synthetic pretraining followed by real-data fine-tuning.Figure 3 diagrams the holistic pipelines used.Source: §3.2 · p. 3, Figure 3 · p. 3
3.3. Label Representation
Label representation balances vocabulary complexity against sequence length. The paper adopts a reversible, decomposed **bekern encoding for full-page transcription.
3.3. Label Representation
- Label encodings trade off vocabulary complexity and sequence length.Graphical primitives are agnostic, while semantic representations such as MusicXML or MEI capture musical meaning but can be too verbose for sequence-to-sequence modeling.Source: §3.3 · p. 3
- The paper uses **bekern, which decomposes **kern tokens into atomic components separated by delimiters.Conversion back to **kern is fully reversible using naive preprocessing tools, preserving compatibility with existing resources.Source: §3.3 · p. 4
- The adopted representation offers compatibility with existing resources through reversible conversion to **kern.The conversion uses naive preprocessing tools.Source: §3.3 · p. 4
3.4. Synthetic Data Generation
The generator builds coherent full-page monophonic scores in typeset and handwritten styles, with separate rendering processes for each style.
3.4. Synthetic Data Generation
- The generator uses RISM incipits that serve as the source for the PrIMuS corpus to pretrain the SMT architecture on synthetic monophonic full-page sheets.It keeps key signature, time signature, and clef constant across staves to avoid notational inconsistencies, then generates pages in either style.Source: §3.4 · p. 4
- Representative generated scores are shown in typeset style in panel (a) and handwritten style in panel (b).The generator aims for visual and syntactic coherence, not genre-specific musical composition.Source: Figure 4 · p. 4, §3.4 · p. 4
3.4.1. Typeset Style
- Typeset pages are rendered with Verovio, with randomized margins, content scale, inter-system spacing, and text elements.Background textures and Augraphy transformations add effects such as ink bleeding, ink shift, and paper texturization to approximate scanned historical prints.Source: §3.4.1 · p. 4
3.4.2. Handwritten Style
- Handwritten pages are rendered with Smashcima from MusicXML converted from the RISM **kern data.The generated image is paired with the **kern sequence as its label sequence.Source: §3.4.2 · p. 4
3.5. Evaluation Corpora
The evaluation uses three real full-page music datasets that vary in layout, writing style, and degradation.
3.5. Evaluation Corpora
- The benchmark comprises three real full-page music datasets with differing layouts, writing styles, and degradation levels.Table 1 reports statistics for the datasets.Source: §3.5 · p. 5, Table 1 · p. 5
- Jazzmus contains 293 handwritten jazz lead sheets gathered from students and professional musicians.The corpus includes irregular handwriting, strike-through corrections, varied capture conditions, and diverse chord-symbol conventions.Source: §3.5 · p. 5
- FMT-C and FMT-M derive from handwritten scores of popular Spanish songs in modern notation.FMT-C has portrait pages with substantial text, whereas FMT-M has landscape pages with paper aging such as stains and yellowing.Source: §3.5 · p. 5
3.6. Evaluation Protocol
The evaluation measures full-page transcription errors with SER and uses ten-fold cross-validation to compare models on limited real datasets.
3.6. Evaluation Protocol
- Symbol Error Rate (SER) measures the minimum insertions, modifications, or eliminations needed to match a predicted symbol sequence to ground truth.The metric is used in previous works.Source: §3.6 · p. 5
- Ten-fold cross-validation addresses the limited size of the real full-page datasets.Results are reported as average SER across folds with standard deviation to capture variability.Source: §3.6 · p. 5
- Dataset examples are shown in the supplementary material because of the resolution of full-page images.Source: §3.6 · p. 5
4. Results
SMT consistently outperformed the baselines across tested corpora regardless of pretraining style. The results also distinguish the effects of pretraining style and show how SMTpr handles errors in a Jazzmus example.
4. Results
- SMT consistently outperformed the other baselines across tested corpora, regardless of pretraining data style.Table 2 reports 10-fold cross-validated SER (%) for FMT-C, FMT-M, and Jazzmus as mean and standard deviation.Source: Table 2 · p. 5, §4 · p. 5
- Across folds, CRNNf and Unfolding were highly unstable, although Unfolding remained competitive with SMThw on FMT-C.Source: §4 · p. 5
- SMTpr pretrained on typeset data had lower SER than handwritten-pretrained SMThw on FMT-C and Jazzmus, with a marginal difference on FMT-M.Source: §4 · p. 5
4.1. Qualitative Evaluation
- On Jazzmus, SMTpr ignores empty staves and the title and resumes after local failures; false negatives are mostly syntactic, while false positives are pitch errors.Source: §4.1 · p. 5, Figure 6 · p. 6
- Because errors span long symbolic sequences across multiple staves, locating them still requires careful manual review.Source: §4.1 · p. 6
5. Conclusions
Across three datasets with differing layout conditions, SMT outperformed segmentation-based and unfolding baselines, which also varied substantially across folds. Typeset pretraining matched or surpassed handwritten pretraining, pointing to page layout and notation conventions rather than handwriting resemblance as the more relevant synthetic-data factors.
5. Conclusions
- Across three datasets with different layout conditions, SMT outperformed segmentation-based and unfolding baselines in 10-fold cross-validation.Source: §5 · p. 6
- The other pipelines had higher error rates and greater variation across folds.Source: §5 · p. 6
- Typeset pretraining matched or surpassed handwritten pretraining on all datasets, with layout and notation conventions mattering more than visual resemblance to handwriting.Source: §5 · p. 6