PaperlayerView exact revision on arXiv

Source-linked AI summary

Statistical Models for Automatic Fingering-Annotated Piano Sheet Music Transcription

Daniel Penner, Abram Hindle

arXiv:2609.28787v1cs.SD

TL;DR

Piano transcription can identify notes, but producing playable sheet music also requires assigning each note to a hand and finger. This paper proposes eight statistical annotation methods and an audio-to-score pipeline; on PIG, Synthesis reaches 90.8% hand accuracy and 56.6% joint accuracy.

  • Problem

    Piano note transcription does not by itself provide hand and finger assignments, and determining the hand from note information alone remains a complex, limited-research problem.

    Source: §I · p. 1, B. Piano Hand Detection · p. 2

  • Method

    The paper proposes eight statistical methods for hand and fingering annotation of transcribed piano notes and develops a pipeline from piano audio to fingering-annotated sheet music.

    Source: §I · p. 1

  • Results

    Synthesis achieves 90.8% hand accuracy and 56.6% combined accuracy, and in the end-to-end pipeline reaches 65.7% hand recall and 37.0% joint recall.

    Source: B. Annotation Accuracy · p. 6, §VI · p. 8

  • Takeaways & Limitations

    The results provide a strong statistical baseline for further research on piano hand and fingering annotation.

    Source: §VI · p. 8

    Evaluations use ground-truth fingerings that may exclude other valid sequences, classical performances for training, and synthesized audio in the end-to-end evaluation rather than real pipeline conditions.

    Source: §V · p. 8

Abstract

from arXiv · show

Machine learning tools have significantly aided automatic piano music transcription; however, this domain has focused primarily on accurately predicting the pitches and timings of played notes. To produce sheet music for the piano, notes must be separated into two staves, one for each hand, and good sheet music often contains fingering annotations to guide the player when sight-reading or learning fast or complex pieces. We propose 8 statistical approaches for combined hand and fingering annotation of transcribed piano notes, including baseline hidden Markov models, rule-based methods, a synthesis of existing approaches, and N-gram language models. Furthermore, we develop a pipeline for complete transcription from piano audio to fingering-annotated sheet music. Evaluations with the PIG dataset demonstrate that our Synthesis model achieves a hand separation accuracy of 90.8\% and a joint hand and finger annotation accuracy of 56.6\%. These approaches serve as a new baseline for further research into this problem, while our pipeline demonstrates the feasibility of a combined system for automated note transcription, hand separation, and fingering annotation.

I. Introduction

The paper addresses the gap between note transcription and playable piano scores annotated with hands and fingers. It also situates this goal among note-transcription methods and work on score generation.

I. Introduction

  • Fingering labels can help players avoid trying multiple fingering paths before practicing or performing a piece.Automatic sheet music can also make unavailable scores accessible and let composers without formal training generate scores from recordings.Source: §I · p. 1
  • Existing transcription systems identify notes accurately but generally produce a single MIDI file or stave, without playable fingerings or an end-to-end combination of these tasks.A complete annotation must assign each note to a hand and finger while accounting for transitions between notes.Source: §I · p. 1
  • The paper proposes eight methods for hand and fingering annotation of transcribed piano MIDI and integrates them into a pipeline from piano audio to annotated sheet music.Experiments use the PIG dataset.Source: §I · p. 1

A. Piano Note Transcription

  • Kong et al.’s transcription system outperformed onsets and frames on all metrics, and the authors use it in their end-to-end pipeline.The system combines convolutional recurrent networks and bidirectional gated recurrent units for velocity, onset, frame, and offset prediction.Source: A. Piano Note Transcription · p. 2
  • Video-based note and hand analysis requires specific pianist video, which is generally inaccessible in transcription tasks.Note-transcription research has increasingly used neural networks and computer vision on dense polyphonic audio or video.Source: A. Piano Note Transcription · pp. 1–2

B. Piano Hand Detection

Hand assignment from note pitch and timing alone remains difficult and has limited research. Related work includes statistical hand tracking and fingering methods adapted from other instruments.

B. Piano Hand Detection

  • Detecting which hand played a note from pitch and temporal information alone is complex, despite several tools that infer hand and finger positions from video.Video-based tools do not establish hand use from note events alone.Source: B. Piano Hand Detection · p. 2
  • Hadjakos et al. proposed Kalman-filter hand assignment from note events, later extending it bidirectionally; the filter is deterministic and computationally inexpensive.A neural-network approach produces slightly more accurate hand estimates, while the Kalman filter offers a statistical alternative.Source: B. Piano Hand Detection · p. 2
  • Nakamura et al.’s merged-output HMMs use two part-HMMs, one per hand, to estimate both hand and finger for note events.The paper reproduces this as a baseline because it was the only existing combined hand-and-finger technique from MIDI described here.Source: B. Piano Hand Detection · p. 2

C. String Instrument Fingering Detection

  • Guitar fingering methods assign costs to chord shapes and transitions, then seek a lowest-cost path through weighted graphs.Maezawa et al. adapted a guitar cost function for violin by adding parameters for violin fingering fundamentals.Source: C. String Instrument Fingering Detection · p. 2

D. Piano MIDI Fingering Detection

Prior piano MIDI fingering work includes HMMs and neural models, while score-generation systems address converting performance data into notation. These approaches vary in their scope and data requirements.

D. Piano MIDI Fingering Detection

  • Yonebayashi et al. model monophonic, single-hand fingering with an HMM using note distances and Gaussian transition costs between fingers.Their representation maps the piano in two dimensions, separating black and white keys and higher and lower pitches.Source: D. Piano MIDI Fingering Detection · p. 2
  • Nakamura et al.’s PIG dataset contains finger annotations for 150 piano pieces across 309 performances, and their higher-order HMMs were more accurate than their standard HMM and DNN approaches.Their chord HMM also models the spread of fingers within near-simultaneous notes played by one hand.Source: D. Piano MIDI Fingering Detection · p. 2
  • Ramoneda et al. report that LSTM and GNN models pretrained on ThumbSet and fine-tuned on PIG outperform Nakamura et al.’s HMMs.Guan et al. combine a bidirectional LSTM with explicit finger-transfer constraints and train on PIG plus HMM-generated data.Source: D. Piano MIDI Fingering Detection · p. 2

E. Score Generation from Transcribed Data

  • Existing audio-to-score pipelines convert detected and quantized notes into MIDI and then traditional scores, while later work uses non-local statistics to choose among possible scores.The cited pipeline combines multipitch detection, rhythm quantization, and score typesetting.Source: E. Score Generation from Transcribed… · pp. 2–3
  • Other score-generation approaches predict notation properties from MIDI, including hand usage, durations, articulations, voice, stem directions, and accidentals.Liu et al. predict beats, note durations, hand separation, and piece metadata; Beyer and Dai use a single transformer for complete score generation.Source: E. Score Generation from Transcribed… · p. 3

III. Proposed Methods

The paper proposes eight hand-and-finger annotation methods, including statistical baselines, a synthesis approach, rule-based methods, and N-gram models. The baseline section details two HMM designs and explains why higher-order HMMs are not used in later evaluations.

A. Annotation Models

  • The eight proposed methods comprise two statistical baselines, a synthesis of existing approaches, two rule-based expansions, and three N-gram approaches.The N-gram methods model annotation as a language problem; the rule-based methods encode constraints of a pianist’s hands.Source: A. Annotation Models · p. 3

1) Baselines

  • Baseline-M uses one HMM whose hidden state jointly represents one of the pianist’s ten hand-finger combinations.Its observations are horizontal and vertical note-to-note distances; initial, transition, and emission probabilities model state sequences and observations.Source: 1) Baselines · p. 3
  • The HMM probabilities are estimated from training counts with smoothing, and Viterbi decoding finds the optimal path.The model uses state, transition, and emission counts.Source: 1) Baselines · p. 3
  • Baseline-S first assigns notes to a hand with a binary-state HMM, then estimates fingers independently with one HMM per hand.Its observations add a low, medium, or high pitch-bin feature, which improved accuracy for this model but had negligible impact on Baseline-M.Source: 1) Baselines · p. 3
  • The PIG dataset’s limited size was judged likely insufficient for training higher-order HMMs without smoothing, so later HMM evaluations use first-order models.The study instead uses smoothed N-gram models with beam search as higher-order alternatives.Source: 1) Baselines · p. 3

2) Kalman-HMM Synthesis Approach

The Synthesis model unifies a bidirectional Kalman filter and an HMM for joint hand-and-finger estimation. The novel rule-based approaches add costs for hand and finger constraints, with Novel-B tracking each hand’s recent state during decoding.

2) Kalman-HMM Synthesis Approach

  • The Synthesis model combines the bidirectional Kalman filter with an HMM in a unified program for training, running, and evaluation.The Kalman filter provides hand separation, while the first-order HMM provides joint hand-and-finger estimation.Source: 2) Kalman-HMM Synthesis Approach · pp. 3–4

3) Novel Rule-Based Approaches

  • Novel-V extends Baseline-M with costs for hand and finger movements, motivated by the risk that a wrong hand assignment degrades nearby finger predictions.Its rules encode repeated-finger use, non-crossing fingers, hand ordering, and ideal finger reach for simultaneous notes.Source: 3) Novel Rule-Based Approaches · p. 4
  • Novel-V can apply weights only from the last state, leaving the other hand’s last state unknown; Novel-B tracks each hand’s last note and uses beam search.Adding the other hand’s pitch to Novel-V’s state space would multiply the possible states by 88.Source: 3) Novel Rule-Based Approaches · p. 4
  • Novel-B uses comfortable ranges for simultaneous-note reaches and sequential finger transitions, with costs tied to transition deviation, timing, and excessive chord stretch.Its decoding state records each hand’s last pitch, finger, and note onset and offset.Source: 3) Novel Rule-Based Approaches · p. 4

4) Novel N-gram Approaches

The N-gram approaches model hand and fingering paths as sequences of tokens derived from hand, finger, and horizontal note-transition distance. They range from a trigram baseline to a four-model combination and a rule-augmented variant.

4) Novel N-gram Approaches

  • N-gram tokens encode hand, finger, and horizontal pitch distance; vertical differences are ignored to limit unique tokens, and the models use interpolated modified Kneser-Ney smoothing.The approach tracks several past states while smoothing unseen combinations.Source: 4) Novel N-gram Approaches · pp. 4–5
  • 3-gram-M uses a single trigram and beam search to find the most likely hand-state path from note-transition distances and previous hand states.It serves as the baseline for the other N-gram approaches.Source: 4) Novel N-gram Approaches · p. 5
  • 3-gram-T combines four trigram models in log-space, including full hand sequences, hand-specific fingering, and hand-only predictions.The hand-specific models use the probabilities of the last n notes assigned to that hand.Source: 4) Novel N-gram Approaches · p. 5
  • 3-gram-R adds non-crossing-finger and simultaneous-note ordering rules to 3-gram-T, subtracting static costs during decoding when a rule is violated.The higher note in a simultaneous pair should receive a higher finger number.Source: 4) Novel N-gram Approaches · p. 5

B. End-to-End Transcription Pipeline

The pipeline transcribes piano audio to MIDI, assigns hand and fingering annotations, and converts the result into sheet music. Its Bach fugue example has generally correct pitches, signatures, tempo, and hand separation, though finger crossings and repetitions remain common.

B. End-to-End Transcription Pipeline

  • The pipeline sends audio-derived MIDI to an annotation model, then rhythm-quantizes the score, replaces PM2S hand estimates, normalizes tempo, and converts the cleaned MIDI to MusicXML.Predicted fingerings are aligned to MusicXML note attacks using hand, pitch, and score onset.Source: B. End-to-End Transcription Pipeline · p. 5
  • The Bach fugue example has generally correct pitches, signatures, tempo, and hand separation, but strange finger crossings or repetitions are common.The authors present it as an example of the potential viability of a unified piano-sheet-music transcription pipeline.Source: B. End-to-End Transcription Pipeline · p. 5

IV. Experimental Methodology

The experiments train and evaluate eight annotation methods on the PIG dataset, then assess a complete pipeline using synthesized audio from PIG pieces. Evaluation uses randomized piece splits and a separately defined end-to-end pipeline test.

A. Experimental Setup

  • PIG contains 150 classical pieces, each with one or more sets of ground-truth hand and finger annotations from real-world performances.Source: A. Experimental Setup · p. 5
  • Nakamura-Merged was recreated from its paper because its implementation code was not publicly available.Source: A. Experimental Setup · p. 5
  • PianoVAM’s fingering labels were not publicly available as of August 2026, so 10 PIG evaluation pieces were synthesized with MuseScore 4 for pipeline testing.PianoVAM contains audio, MIDI notes, and fingering annotations, which would otherwise allow evaluation of each pipeline step.Source: A. Experimental Setup · p. 5
  • 100 randomized 130/10 train/test splits yielded 1,000 piece-level evaluations; pipeline evaluation used 10 synthesized PIG files as input.The final 10 PIG pieces were reserved for informal development tests.Source: A. Experimental Setup · p. 5
  • The pipeline diagram outlines stagewise data flow, and a sample Bach score was generated entirely by the pipeline using 3-gram-R for annotation.Source: Figure 1 · p. 6, Figure 2 · p. 6
  • All methods used the same 100 randomized splits; N-gram beam size was 50 and Novel-B’s was 20, and accuracy used Nakamura et al.’s recombined match rate.Novel-B’s smaller beam size reflected its slower runtime; models and evaluation scripts were available on Zenodo.Source: A. Experimental Setup · p. 6

B. Annotation Accuracy

Synthesis achieved the strongest model-only annotation accuracy and the strongest end-to-end pipeline recall among the evaluated methods. The pipeline comparison uses a lenient onset tolerance to emphasize annotation correctness.

B. Annotation Accuracy

  • 90.8% hand accuracy and 56.6% joint accuracy made Synthesis the strongest model on both annotation measures and the lowest-runtime model.3-gram-R was second on both measures, at 90.4% hand accuracy and 50.7% joint accuracy.Source: B. Annotation Accuracy · p. 6

C. Transcription Pipeline Recall

  • Paired bootstrap tests found significant differences in annotation accuracy for 35/36 model pairs and in pipeline hand and joint recall for 34/36 and 33/36 pairs, respectively.Insignificant pairwise differences are reported in Table 3.Source: B. Annotation Accuracy · p. 7, C. Transcription Pipeline Recall · p. 7, Table 3 · p. 7
  • 100 randomized evaluation splits compared model annotations against ground-truth hand and finger labels, while pipeline recall also checked note pitches and onsets.The pipeline evaluation used a 250 ms onset tolerance to focus on annotation correctness.Source: C. Transcription Pipeline Recall · p. 7
  • 65.7% hand recall and 37.0% joint recall made Synthesis the strongest end-to-end model; 3-gram-R reached 64.9% and 33.6%, respectively.A note counted as recalled when its pitch was correct, its onset was within 250 ms, and its hand or joint annotations were accurate.Source: C. Transcription Pipeline Recall · p. 7, Table 4 · p. 7

D. Pipeline Comparison

The pipeline comparison finds Audio-to-Score ahead on transcription and hand separation for a limited released subset, while the paper’s pipeline also provides fingering annotation. In a separate out-of-dataset example, 3-gram-R avoids three specified error types that appear in the Synthesis output.

D. Pipeline Comparison

  • Audio-to-Score outperformed this pipeline in transcription and hand separation on the released MAPS subset, but results were comparable and the subset was small.Unlike Audio-to-Score, this pipeline also enables fingering annotation and is open-source for use on any audio file.Source: D. Pipeline Comparison · p. 7

E. Qualitative Example

  • The qualitative comparison added only predicted finger labels to manually transcribed staves, notes, rests, ties, and time signature.Notes were supplied as one stream regardless of hand or stave; positive integers represented right hand and negative integers left hand.Source: E. Qualitative Example · pp. 7–8
  • On the selected Chopin passage outside the dataset, Synthesis had nine instances of the specified error types, while 3-gram-R avoided all three types.The types were quick successive reuse of the same finger, unplayable finger crossings, and incorrect hand annotations.Source: E. Qualitative Example · p. 8, Figure 3 · p. 8, Figure 4 · p. 8

V. Threats to Validity

The conclusion reports Synthesis annotation and end-to-end results, while the validity discussion identifies label, training-genre, and audio-evaluation constraints.

V. Threats to Validity

  • Ground-truth fingering labels may falsely invalidate estimates because they do not represent the only possible fingering sequence.Source: §V · p. 8
  • Classical-only training may limit the models’ viability for other genres or musical styles.The authors note that the models are purely statistical.Source: §V · p. 8
  • End-to-end evaluations used synthesized piano audio, unlike the transcription model’s real-recording training data, potentially reducing the accuracy of notes passed to annotation.The authors say this setup does not reflect real pipeline conditions.Source: §V · p. 8

VI. Conclusion

  • 90.8% hand-separation accuracy and 56.6% joint hand and fingering accuracy for Synthesis, versus 90.4% and 50.7% for 3gram-R.End-to-end Synthesis achieved 65.7% hand recall and 37.0% joint recall.Source: §VI · p. 8
  • The authors identify advanced joint-estimation models and larger, more comprehensive datasets as next research steps.They say such work could make proper end-to-end transcription possible.Source: §VI · p. 8
Loading 2609.28787v1…