PaperlayerView exact revision on arXiv

Source-linked AI summary

Understanding Optical Music Recognition

Jorge Calvo-Zaragoza, Jan Hajič, Alexander Pacha

arXiv:1908.03608v3cs.CVcs.AIcs.IRcs.SDeess.AS

TL;DR

OMR lacks accessible introductory material and shared terminology, while its overall evaluation remains under-specified because OMR serves diverse applications. The paper defines OMR, relates it to neighboring fields, analyzes it as an inverse music-encoding process, proposes application taxonomies, and discusses modern deep-learning approaches. It concludes that OMR systems must be designed and evaluated for specific goals and applications.

  • Problem

    OMR has lacked accessible introductory material, shared terminology, and sufficiently specified evaluation questions for its diverse applications.

  • Method

    The paper defines OMR, relates it to neighboring fields, analyzes its inverse relationship to music encoding, proposes taxonomies, and describes a traditional processing pipeline.

  • Results

    The analysis shows that OMR is not a monolithic task and that different use cases require fundamentally different system designs and evaluation approaches.

  • Takeaways & Limitations

    OMR research should specify the intended application and goals before designing or evaluating a system.

  • Takeaways & Limitations

    OMR errors can make transcription less useful than direct note entry, while acceptable error levels depend on the application and remain inconclusive in existing literature.

Abstract

from arXiv · show

For over 50 years, researchers have been trying to teach computers to read music notation, referred to as Optical Music Recognition (OMR). However, this field is still difficult to access for new researchers, especially those without a significant musical background: few introductory materials are available, and furthermore the field has struggled with defining itself and building a shared terminology. In this tutorial, we address these shortcomings by (1) providing a robust definition of OMR and its relationship to related fields, (2) analyzing how OMR inverts the music encoding process to recover the musical notation and the musical semantics from documents, (3) proposing a taxonomy of OMR, with most notably a novel taxonomy of applications. Additionally, we discuss how deep learning affects modern OMR research, as opposed to the traditional pipeline. Based on this work, the reader should be able to attain a basic understanding of OMR: its objectives, its inherent structure, its relationship to other fields, the state of the art, and the research opportunities it affords.

I. INTRODUCTION

This tutorial makes Optical Music Recognition more accessible by defining its scope, clarifying its terminology and applications, and organizing traditional and deep-learning approaches. It frames OMR as a research field that computationally reads music notation in documents, while distinguishing notation from the musical information it encodes.

  • Motivation: The field has remained fragmented because its goals, terminology, and motivating applications were not defined rigorously enough for researchers and external stakeholders to follow its state of the art.The paper attributes this opacity to the fusion of music-centric and document-centric perspectives.
  • Applications and evaluation: OMR applications differ substantially, so claims about whether OMR works must specify the intended application rather than treating the field as one monolithic problem.The paper also distinguishes tasks with straightforward evaluation from genuinely difficult cases such as full score re-typesetting.
  • Conceptual framework: OMR separates the visual language of music notation from the musical information it encodes, and its definition intentionally leaves output representation unspecified.Different tasks require different outputs, so fixing one representation would make the definition unnecessarily restrictive.
  • Contributions: The tutorial systematizes OMR through a robust definition, terminological clarifications, an applications review, evaluation guidance, and a bibliography.Its stated novelty is collecting and organizing fragments from existing literature to make the field easier to approach and collaborate on.
  • Relationship to related fields: The paper relates OMR to OCR and other fields while emphasizing that OMR computationally reads notation rather than studying notation systems or human musical interpretation.This boundary supports a field-level definition that remains inclusive of different tasks and outputs.
  • Definition and scope: OMR is defined as a research field that investigates how to computationally read music notation in documents.The definition distinguishes the field from individual tasks and from studying notation systems or human reading.

III. FROM “MUSIC” TO A DOCUMENT

Music notation encodes structured musical ideas through visual symbols, rules, and layout decisions in a document. OMR treats this embodied notation as the result of an encoding process that can be computationally inverted.

  • Music notation systems use symbols and positioning rules to preserve key musical properties, while some information may be under-specified or omitted.Pitches, durations, and onsets are preserved especially accurately, whereas aspects such as tempo may not be fully captured.
  • Common Western Music Notation is the most prominent modern notation system, but historical notations and other systems also encode music visually.CWMN evolved from mensural notation and stabilized at the beginning of the nineteenth century.
  • A single set of notes can be engraved in many ways, because spacing, voice consistency, clefs, and page layout affect readability and logical structure.Engraving decisions may have local effects, such as object spacing, or global effects, such as page-break placement.
  • Music is modeled as a structured assembly of notes enriched with performer instructions and encoded visually in a notation system and medium.Notes are characterized by pitch, duration, loudness, timbre, and onset; notation also encodes instructions such as bowing, fingerings, and articulations.

IV. INVERTING THE MUSIC ENCODING PROCESS

OMR begins with a music notation document and can be understood as inverting the process that encoded musical information into visual form. It distinguishes recovering notation from recovering musical semantics, while excluding human interpretation beyond what the document specifies.

  • OMR starts after a musical composition has been expressed visually with music notation in a document.In this context, the document is usually understood as a digital image, though alternatives are discussed elsewhere.
  • Reading a music document involves identifying visual symbols and configurations, decoding notes and instructions, and interpreting information that notation leaves unspecified.The final interpretive step uses musical intuition, prior knowledge, and taste and is explicitly outside OMR.
  • OMR separates recovering the notation and engraving information from recovering musical semantics such as pitches, velocities, onsets, and durations.Notation recovery corresponds to the visual arrangement and may use MusicXML or MEI, whereas semantic recovery may use MIDI.
  • Recovering complete notation can make musical semantics unambiguous, but recovering semantics alone does not necessarily restore how notes were arranged in the engraving.The two recovery goals are closely related but are not interchangeable.

A. Alternative Names

OMR is a computational research field concerned with reading music notation in documents, not merely a generic form of OCR. Its distinctiveness arises from notation’s contextual visual language and the need to recover musical semantics.

  • Alternative Names: OMR is positioned within computer vision and document analysis, while its outputs support Music Information Retrieval and searchable digital music libraries.The field also connects to deep learning as a source of new approaches and to practical score digitization.
  • Optical Music Recognition vs. Text Recognition: Music notation is a contextual writing system whose symbols acquire meaning through their arrangement on staves and relationships to one another.Primitive symbols such as stems, noteheads, and flags have clear interpretations, but configurations specify what notes should be played.
  • Optical Music Recognition vs. Text Recognition: OMR must often recover musical semantics, such as pitch, in addition to recognizing graphical symbols, unlike typical text recognition.A note’s graphical position is interpreted together with clefs and accidentals to determine its effective pitch.
  • Optical Music Recognition vs. Text Recognition: Music notation differs from ordinary writing through highly variable primitive sizes, scale and rotation changes, flexible curves, and pervasive two-dimensional spatial relationships.These properties are illustrated in Beethoven’s score excerpt.

B. Optical Music Recognition vs. Other Graphics Recognition Challenges

Music notation presents graphics-recognition challenges through content-dependent alignment, simultaneous events, complex semantics, and rule flexibility. The paper therefore systematizes OMR around applications and tasks rather than reviewing only technical methods.

  • Optical Music Recognition vs. Other Graphics Recognition Challenges: Content-dependent typographical alignment can shift simultaneous notes to avoid overlaps, making visual position diverge from musical simultaneity.Examples include chord notes and moving eighth notes that are played together but forced to different horizontal positions.
  • Optical Music Recognition vs. Other Graphics Recognition Challenges: Music notation has complex semantics and implicit rules, including duration constraints that researchers have incorporated into OMR post-processing.Grammatical methods have been used to represent some of these rules.
  • Optical Music Recognition vs. Other Graphics Recognition Challenges: Existing grammatical approaches still lack tractable two-dimensional parsing algorithms powerful enough to recognize music notation without fragile segmentation heuristics.The paper leaves open whether parsers for mathematical expressions can be applied to music notation.
  • Optical Music Recognition vs. Other Graphics Recognition Challenges: The paper proposes systematizing OMR by motivating applications, subtasks, and interfaces, with the expectation that natural task groups can share evaluation protocols.This objective differs from reviews focused primarily on technical solutions.

A. OMR Inputs

OMR inputs vary by notation system, visual and structural complexity, writing mode, and image quality. These dimensions determine the constraints and requirements imposed on recognition systems.

  • Input modality: Offline OMR processes static images, whereas online OMR captures notation as it is written and benefits from stroke-level over-segmentation.Online recognition is also called pen-based recognition in the handwritten recognition community.
  • Notation systems: Music can be represented in diverse notation systems, including Common Western Music Notation, historical notations, tabulature, and Braille.The Standard Music Font Layout lists over 2440 recommended characters plus several hundred optional glyphs.
  • Structural complexity: Structural complexity ranges from monophonic and homophonic music to polyphonic and pianoform scores with interacting staves and voices.Pianoform scores cannot be disassembled into a series of monophonic scores, making their encoded music substantially more complex.
  • Image quality: Image quality depends independently on the underlying document condition and the digital acquisition mode.Inputs range from near-flawless born-digital or printed documents to degraded manuscripts and distorted smartphone photographs.

B. OMR Outputs

OMR outputs should be organized by the application goal and required comprehension, separating metadata extraction, search, replayability, and structured encoding. This taxonomy clarifies evaluation because different outputs require different representations and assessment strategies.

  • Evaluation: OMR evaluation remains difficult because symbol-level scores may not generalize, commercial availability is ambiguous evidence, and no meaningful score edit distance is known.The paper argues that “Does OMR work?” becomes answerable only when tied to a specific application, while task definitions must remain broad enough for comparison.
  • Replayability and structured encoding: Replayability recovers pitch, velocity, onset, and duration so machines can process or play symbolic music without requiring human-readable notation.A MIDI-like representation is typically sufficient for replayability, whereas human readability requires structured encoding.
  • Replayability and structured encoding: Structured encoding recovers both the music and the notation elements used to represent it, supporting readable score output.The representation may use formats such as MusicXML or MEI, which can be complex.
  • Application taxonomy: The proposed application taxonomy orders Document Metadata Extraction, Search, Replayability, and Structured Encoding by increasing comprehension of music notation.Metadata extraction may require minimal understanding, while structured encoding requires complete understanding of notation intricacies.
  • Application taxonomy: Output categories matter because applications differ in output representation and therefore support different evaluation strategies.The taxonomy is intended to avoid comparing results from fundamentally different application goals.

1) Document Metadata Extraction:

Document metadata extraction answers questions about a notation document rather than reconstructing all of its musical content. Its scalar or categorical outputs enable straightforward classification or regression evaluation.

  • Definition and outputs: Document metadata extraction answers questions about a music notation document using scalar values or category labels.Examples include identifying the notation, estimating the writing period, counting instruments, and identifying a copyist.
  • Examples: Metadata applications can classify whether images depict music scores and identify document writers, supporting large-scale cataloging and analysis.The cited systems use both traditional computer vision and deep convolutional neural networks.
  • Evaluation: Metadata extraction has unequivocal evaluation protocols because classification and regression tasks use established metrics such as accuracy, F-measure, and mean squared error.The output type does not imply that extracting the target value is easy.
  • Relation to search: Search applications retrieve collection items relevant to a musical query, while metadata extraction instead produces a document-level category or value.Search outputs may identify a complete piece, page, bounding box, or pixel-level location and can be evaluated with precision, recall, or average precision.
  • Search methods and limitations: OMR-based search can transform documents into symbolic information for classical retrieval, but semantic recognition errors become critical when queries or databases contain true musical semantics.When queries and documents are projected through the same OMR system, some limitations may cancel out.

3) Replayability:

Replayability applications reconstruct enough encoded notes to create an audible version of written music. They commonly target MIDI-like symbolic sequences, although recognition can ignore notation elements irrelevant to playback.

  • Definition and uses: Replayability recovers pitches, velocities, onsets, and durations, usually as a MIDI-like symbolic representation that can support audio synthesis and downstream music tools.Producing the actual audio file is not itself considered part of OMR.
  • Applications: Replayability can support practice assistance, symbolic music-information retrieval, large-scale music-theoretical analysis, and creativity-focused applications.MIDI has a long tradition of computational processing and is therefore a key target representation.
  • Digital musicology: Replayability from manuscripts could make previously unrecorded and untypeset compositions available for quantitative musicological research.The paper presents this as a potential route to studying broader historical patterns in musical style.
  • Recognition strategy: Replayability-oriented systems traditionally recover structured encoding first, but they can simplify recognition by ignoring notation elements that do not affect the note sequence.Some systems focus directly on duration-pitch pairs in single-staff images regardless of how those notes are expressed visually.
  • Evaluation: MIDI-like outputs can be evaluated by comparing pitch-onset-duration triplets, with velocity comparisons treated as optional or secondary after note correspondence is established.Proposed alternatives include melodic similarity metrics, multi-pitch estimation metrics, Dynamic Time Warping, and edit distances over ordered pitch-duration sequences.

4) Structured Encoding:

Structured encoding aims to recover a score’s complete musical content together with how notation encodes and lays it out. It remains difficult because suitable representations and reliable evaluation methods are lacking.

  • Structured encoding targets human-readable score digitization and manipulation tasks such as exporting, merging, or transposing parts.
  • Structured Encoding fully decodes musical content while retaining information about how it was encoded through notation.
  • Recovering structured encoding is harder than representing music as time-stamped events because it must reconstruct notation comparable to a structured document description.
  • No automatic method reliably evaluates structured encoding, partly because music-notation representations and formal models remain insufficient.
  • MusicXML and MEI are useful output formats, but they are not suitable as internal OMR representations because they describe scores top-down.

VII. APPROACHES TO OMR

OMR traditionally decomposes recognition into preprocessing, object detection, notation assembly, and encoding, while deep learning has improved or collapsed several stages. Alternative end-to-end and interactive approaches broaden how the process is designed.

  • The traditional OMR pipeline consists of preprocessing, music object detection, notation assembly, and encoding.
  • Deep learning has substantially improved tasks such as staff-line removal and symbol classification, making some pipeline steps obsolete or clearly solvable.
  • End-to-end systems attempt to address the whole recognition process in a single step, but fully inferring musical semantics remains difficult.
  • Machine learning has advanced OMR alongside public datasets and annotation tools, but shifted substantial costs toward acquiring training data.
  • Interactive systems incorporate user feedback because OMR errors may remain unavoidable and some applications cannot tolerate them.

VIII. CONCLUSIONS

The paper defines OMR, systematizes its outputs and applications, and updates its account of methods in light of machine learning. Its central conclusion is that OMR comprises diverse tasks with different goals, challenges, and error tolerances.

  • OMR is defined as a research field investigating how to computationally read music notation in documents and relate it to neighboring fields.
  • The paper proposes a taxonomy of OMR outputs and applications, addressing a previously unsystematized part of the field.
  • The paper updates OMR’s subtask breakdown to reflect machine learning and discusses end-to-end and interactive approaches.
  • OMR is not a monolithic task: different use cases require fundamentally different system designs and associated challenges.
  • Error tolerance depends on the end user and application, because correction effort can determine whether an OMR system is useful.

A. Open Issues and Perspectives for Future Research

The paper identifies open technical and non-technical problems that limit OMR’s progress, including legal and community barriers, missing standards, weak evaluation, and difficult recognition stages. Future systems must also address generalization, interaction, and full end-to-end recognition.

  • OMR lacks standard output representations, causing incompatibilities between systems and making subcomponent replacement difficult.
  • Evaluation remains unsatisfactory because OMR outputs lack standards and ideal intrinsic metrics are not yet established.
  • Structured encoding remains a major open problem because few scientific systems pursue reconstruction of the input’s full structure.
  • Full end-to-end recognition remains difficult because music notation has complex outputs unlike the one-dimensional sequences handled by common recurrent networks.
  • Generalizing machine-learning systems to real-world collections is pressing because data acquisition remains a significant bottleneck.
  • Interactive OMR includes users when errors cannot be tolerated, shifting the objective toward reducing user effort rather than only improving accuracy.

APPENDIX A: OMR BIBLIOGRAPHY

The paper accompanies a curated, open bibliography of OMR research and related publications, assembled from multiple sources and manually verified against trustworthy references.

  • The authors publish a comprehensive, curated OMR bibliography in an open-source GitHub repository that accepts submissions through pull requests and templated issues.The website is automatically generated from BibTeX files using BibTex2HTML.
  • The repository separates OMR research publications, related publications, and other bibliographic material into distinct files rendered as separate pages.The OMR Research Bibliography contains scientific and technical publications with manually verified metadata, usually including a DOI or publication link.
  • The bibliography merges historical collections from multiple researchers with a recent extensive BibTeX library and ongoing discovery through a Google Scholar Alert.The alert tracks citations to the survey by Rebelo et al., described as a widely cited state-of-the-art review.
  • Entries are verified by searching for publication details and checking trustworthy sources such as original publishers, author websites, venue websites, and major indexing services.Google Scholar is used to locate works before verification against sources including IEEE Xplore, ACM Digital Library, Springer Link, ScienceDirect, arXiv, DBLP, or ResearchGate.
Loading 1908.03608v3…