PaperlayerView exact revision on arXiv

Source-linked AI summary

DuET: Note-Frame Dual Encoding for Piano Performance-to-Score Transcription

Wei Zeng, Ye Wang

arXiv:2610.21467v1cs.SDcs.LGeess.AS

TL;DR

Expressive piano-to-score transcription must interpret note timings within a metrical structure, although beats may not coincide with note onsets. DuET jointly models note-level PM2S and frame-level beat/downbeat tracking with shared time and time-pitch encodings. It achieves the lowest reported MUSTER mean error among compared methods on ASAP and Gangqinpu, with improvements in timing-related metrics and a substantial overall-error reduction on Gangqinpu.

  • Problem

    Expressive performance-to-score transcription must map note timings within metrical context, even though beat positions can occur on frames without note onsets.

    Source: §1 · p. 1

  • Method

    DuET jointly learns note-level PM2S transcription and frame-level beat/downbeat tracking, connecting the branches with cross-attention and encoding relative performance time and pitch.

    Source: §1 · p. 2

  • Results

    DuET achieves the lowest MUSTER mean error among compared methods on ASAP and Gangqinpu, with consistent timing-metric improvements and a substantial overall transcription-error reduction on Gangqinpu.

    Source: §1 · p. 2, §6 · p. 11

  • Takeaways & Limitations

    Ablations demonstrate the value of the frame branch and decoder-history masking, while benefits of pitch-aware encoding and memory integration vary across datasets and metrics.

    Source: §6 · p. 11

    The one-to-one mapping excludes note insertion, deletion, and many-to-one ornament reduction during prediction.

    Source: §3.1 · p. 4

Abstract

from arXiv · show

Transcribing expressive piano performances into scores requires interpreting individual note timings within an underlying metrical structure. However, score attributes are associated with notes, whereas beats may occur without coincident note onsets. We introduce DuET, a Dual-Encoder Transformer that jointly learns note-level performance MIDI-to-score (PM2S) transcription and frame-level beat and downbeat tracking. Bidirectional cross-attention connects note and frame encoders, while an autoregressive decoder alternates between their representations to predict structured score attributes. We further introduce TimeRoPE and Time-Pitch Rotary Positional Encoding (TPRoPE) to establish a shared performance-time reference and explicitly encode relative temporal and pitch relationships. Evaluations on ASAP, a classical piano dataset, and our curated Gangqinpu dataset, which spans multiple genres with a focus on popular piano music, demonstrate improvements in score timing and notated duration. Compared with the strongest baseline on each dataset, DuET reduces MUSTER mean error from 6.56% to 3.64% on Gangqinpu and from 11.30% to 11.22% on ASAP. Ablations support the benefits of frame-level context, time-based positional encoding, and decoder-history masking. Code, weights, and demos are available at https://github.com/wei-zeng98/DuET.

1 Introduction

PM2S transcription must map expressive note timings into score time while using metrical context that may fall between note onsets. DuET addresses this with joint note- and frame-level modeling and time/pitch encodings, and achieves the lowest reported MUSTER mean error among compared methods on both evaluated datasets.

1 Introduction

  • Accurate score transcription requires interpreting note timings within metrical context because the same performed interval can map to different score-time intervals under expressive timing.Beat positions occupy a frame-level grid and may occur without a coincident note onset.Source: §1 · p. 1
  • Prior neural PM2S work has primarily used classical datasets, motivating evaluation across classical and popular piano music.The paper notes that sheet music is often less readily available for popular piano music.Source: §1 · p. 2
  • PM2S maps performance MIDI to notated scores, predicting score onsets, durations, staff and voice assignments, meter and key signatures, and grace-note labels.The paper focuses on this intermediate stage of an A2S pipeline, using Transkun v2 for audio-to-performance MIDI and MuseScore 4 for score formatting.Source: §1 · p. 2
  • DuET jointly models note-level PM2S transcription and frame-level beat/downbeat tracking, linking the branches through bidirectional encoder and alternating decoder cross-attention.This combines note relationships with metrical context and supports supervision at both prediction granularities.Source: §1 · p. 2
  • DuET achieves the lowest MUSTER mean error and the lowest onset, offset, duration, and staff-assignment errors among compared methods on both ASAP and Gangqinpu.Gangqinpu spans multiple styles with a focus on popular piano music; ASAP contains classical piano performances.Source: §1 · p. 2
  • TimeRoPE establishes a shared performance-time reference, while TPRoPE encodes relative temporal distances and pitch intervals in note attention.These encodings address the limits of sequence order as a representation of relative time and pitch.Source: §1 · p. 2

2 Related work

Prior PM2S systems have incorporated metrical structure through note-based inference, implicit score positions, or externally supplied beat grids, while evaluation has largely centered on classical piano. Positional encodings have likewise emphasized sequence relationships; DuET adapts multidimensional rotary encoding to performance time and pitch.

2.1 Piano performance MIDI-to-score transcription

  • DuET jointly learns frame-level beat tracking and note-level score prediction, avoiding the need for an externally supplied beat grid at inference.Earlier approaches infer beats without note onsets, model meter implicitly, or condition quantization on a supplied grid whose errors can propagate.Source: §2.1 · p. 3
  • DuET complements prior classical-focused PM2S evaluations with training and evaluation on both classical and popular piano music.Prior work also examined popular piano, and Wachter et al. explored jazz guitar, but its piano evaluation remained classical.Source: §2.1 · p. 3

2.2 Positional encoding in symbolic music

  • TPRoPE adapts multidimensional rotary encoding to symbolic music, using time and pitch coordinates for note attention and a shared performance-time reference across note and frame representations.Earlier positional encodings primarily represented sequence-index relationships rather than both performance-time distances and pitch intervals.Source: §2.2 · p. 3

3 Methodology

DuET formulates performance MIDI-to-score transcription as note-aligned score prediction, with frame-level beat tracking to represent metrical context beyond note onsets. Its dual encoders and time- and pitch-aware positional encodings support score prediction, which is performed across contextual windows and reconstructed into a full score.

3.1 Task formulation

  • DuET predicts score attributes for a one-to-one, order-preserving mapping from input performance notes, while beat and downbeat labels provide separate frame-level metrical supervision.The mapping excludes note insertion, deletion, and many-to-one ornament reduction during prediction.Source: §3.1 · pp. 3–4

3.2 DuET architecture

  • Eight coupled note and frame encoder layers exchange information through bidirectional cross-attention, and an autoregressive decoder alternates between their memories to predict score records.The note branch models notes and their relationships; the frame branch captures temporal context on a regular grid.Source: §3.2 · pp. 4–6, Figure 1 · p. 5

3.3 TimeRoPE and TPRoPE

  • TimeRoPE uses shared performance-time coordinates, while TPRoPE encodes relative time and pitch for note attention; IndexRoPE represents decoder record order.TimeRoPE uses continuous note onsets without rounding to the 20 ms frame grid.Source: §3.3 · pp. 6–7, Figure 2 · p. 7

3.4 Windowing and continuation

  • During training and inference, 32-second new-note regions use preceding performance context and up to four bars of score-prefix context; inference appends each note prediction once.Only new records receive note-level training losses, and inference windows advance through non-overlapping new regions.Source: §3.4 · p. 7

3.5 Score reconstruction

  • Full-score reconstruction selects the most frequent bar-length, time-signature, and key-signature tuple that accommodates predicted within-bar onsets, with empty bars inheriting preceding signatures.Bar starts are accumulated from bar lengths, and MuseScore 4 converts the structured score into notation.Source: §3.5 · pp. 7–8

4 Experimental setup

Training and evaluation use (n)ASAP and Gangqinpu, with additional training data from PERiScoPe; piece-level splits prevent overlap. The setup also specifies note and decoder-history masking and the model’s training configuration.

Datasets

  • Gangqinpu contains 18,787 performance–score pairs spanning popular, classical, and other styles, while (n)ASAP focuses on Western classical repertoire.PERiScoPe provides additional training data; all three datasets’ performances were transcribed using Transkun v2.Source: §4 · p. 8
  • Piece-level train, validation, and test splits use cross-dataset work groups to prevent overlap.Note correspondences and metrical supervision support joint score prediction and beat tracking.Source: §4 · p. 8

Masking

  • Training masks selected bars’ note content with probability 0.1 and shifted decoder-history embeddings with probability 0.75; masking is disabled at evaluation and inference.Current-note pitch embeddings are also masked for selected bars, while original onset and pitch coordinates for RoPE are retained.Source: §4 · p. 8

Training configuration

  • DuET trains for 50 epochs from random initialization using AdamW, a peak learning rate of 2 × 10−4, and weight decay of 0.01.Training uses two GPUs, a global batch size of 32, cosine decay, and linear warmup over the first 5% of training steps.Source: §4 · p. 8

5 Results

DuET has the lowest reported MUSTER mean error on both test datasets, with gains in score timing and notated duration. Ablations show that performance depends on frame context, positional encoding, decoder memory access, and masking; a qualitative example contrasts DuET’s rhythmic organization with two baselines.

5.1 PM2S transcription performance

  • 3.64% versus 6.56% MUSTER mean error on Gangqinpu and 11.22% versus 11.30% on ASAP give DuET the lowest reported error on both datasets.Gangqinpu is compared with the strongest baseline; ASAP is compared with the published Transformer result.Source: §5.1 · p. 8, Table 2 · p. 9
  • DuET also has the lowest errors on all four reported ScoreSimilarity metrics and lower onset, offset, duration, and staff-assignment errors on ASAP, but higher missing-note and extra-note errors than several baselines.Across both datasets, the most consistent gains are in score timing and notated duration.Source: §5.1 · p. 8

5.2 Ablations

  • Removing the frame branch raises MUSTER mean error from 8.68% to 10.50% on ASAP and from 3.07% to 3.64% on Gangqinpu.Onset and offset errors also increase on both datasets; duration results are mixed.Source: §5.2 · p. 9, Table 3 · p. 9
  • With alternating memory access, Time–Pitch RoPE lowers MUSTER mean error from 13.68% to 8.68% on ASAP and from 3.57% to 3.07% on Gangqinpu.Its effect with single-memory decoding is mixed and depends on decoder memory and dataset.Source: Table 3 · p. 9, §5.2 · p. 10
  • With Time–Pitch RoPE, alternating access yields lower MUSTER mean error than note-only or frame-only memory on both datasets; with time-only RoPE, single-memory variants do better.Several single-memory variants also have lower duration errors than DuET.Source: Table 3 · p. 9, §5.2 · p. 10
  • Removing encoder-input bar masking or decoder-history note masking raises MUSTER mean error on both datasets; removing decoder-history masking also substantially raises onset error.The combined masking setting does not minimize every metric: removing input masking improves Gangqinpu note-deletion and note-insertion errors, while some variants achieve higher beat F1.Source: Table 3 · p. 9, §5.2 · p. 10, Table 4 · p. 10

5.3 Qualitative analysis

  • In a randomly selected excerpt, DuET produces regular 4/4 meter and simpler note values, while the two baselines introduce meter changes, tuplets, fragmented rests, or overlapping voices.The authors describe DuET’s excerpt as more coherent and readable, while cautioning that visual simplicity alone does not establish accuracy.Source: §5.3 · pp. 10–11, Figure 3 · p. 11

6 Conclusion

DuET combines note-level score prediction with frame-level metrical supervision and improves timing-related metrics on ASAP and Gangqinpu, with a substantial reduction in overall transcription error on Gangqinpu.

6 Conclusion

  • DuET shows consistent improvements in timing-related metrics and substantially reduces overall transcription error on Gangqinpu.Experiments cover ASAP and Gangqinpu.Source: §6 · p. 11
  • Bidirectional encoder and alternating decoder cross-attention integrate note relationships with temporal context for PM2S transcription.The system combines note-level score prediction with frame-level metrical supervision.Source: §6 · p. 11
  • TimeRoPE and TPRoPE represent relative performance time and pitch intervals.Source: §6 · p. 11
  • Ablations show value from the frame branch and decoder-history masking, while pitch-aware encoding and memory integration vary across datasets and metrics.Future work targets downbeat transcription accuracy and evaluation across broader musical styles.Source: §6 · p. 11
Loading 2610.21467v1…