Source-linked AI summary
A Learned Musical-Time Coordinate for Piano Audio-to-Score Transcription
Min-Hsing Wang, Tsung-Ren Huang
TL;DR
End-to-end piano A2S systems mainly handle pre-segmented excerpts, while whole-piece transcription without external segmentation remains limited. The paper jointly learns downbeat phase with score transcription and uses it to guide decoding and derive bar boundaries. Evaluation suggests improved metrical structure and whole-piece transcription with comparable or better notation quality.
Problem
Prior end-to-end A2S systems mainly transcribe pre-segmented excerpts, and how to assemble segments into whole pieces remains mostly unexplored.
Source: §1 · p. 1Method
Results
Takeaways & Limitations
Bar boundaries derived from learned downbeat phase let the system stitch segments into whole pieces without an external beat tracker.
Source: §4 · p. 4Piano-A2S accepts 4–12 s inputs, so longer annotated segments reduce transcription coverage.
Source: §4 · p. 4
Abstract
from arXiv · showhide
End-to-end piano audio-to-score (A2S) transcription is the task of reconstructing a notated score from an audio recording in a single system. Existing A2S approaches usually rely on pre-segmented excerpts, and few handle whole pieces without external segmentation. This raises the question of whether a musical-time coordinate can guide score transcription while also helping the system segment whole pieces. Previous work on jazz saxophone A2S uses an external beat tracker to obtain such a coordinate for score prediction. We introduce downbeat phase, a musical-time coordinate learned jointly with score transcription that represents each audio frame's relative position within a notated bar. With this coordinate, our system can not only condition its score decoder on downbeat phase but also self-segment at bar boundaries and stitch the segments into a whole piece. Evaluation suggests that the coordinate improves metrical structure and supports whole-piece transcription with comparable notation quality.
1. INTRODUCTION
Existing end-to-end A2S systems mainly transcribe excerpts already segmented at bar boundaries, leaving assembly into whole pieces largely unexplored. This work learns downbeat phase jointly with piano score transcription, using it to guide decoding and to self-segment and stitch whole pieces.
1. INTRODUCTION
- Prior end-to-end A2S systems mainly predict score sequences from excerpts pre-segmented at bar boundaries, while assembling segments into whole pieces remains mostly unexplored.The cited systems include Piano-A2S and a Transformer approach for polyphonic string music.Source: §1 · p. 1
- Earlier jazz saxophone A2S used externally estimated beat phase for score guidance, leaving open whether the coordinate itself could be learned jointly with transcription.Source: §1 · p. 1
- The system learns downbeat phase jointly with score transcription, conditions its decoder on that coordinate, and uses its phase posterior to self-segment and stitch whole pieces.The coordinate is supplied through Fourier positional encoding.Source: §1 · p. 1
- Evaluation suggests the learned coordinate helps capture metrical structure and supports whole-piece transcription with comparable or better notation quality, without external segmentation.Source: §1 · p. 1
2. METHOD
The system learns downbeat phase alongside score transcription and uses it to connect audio frames with score time. Its method combines phase-aware decoding, joint training, and bar-based inference to assemble whole-piece scores.
2. METHOD
- The system processes log-mel audio windows and autoregressively generates a **kern token sequence encoding pitches, note values, signatures, bars, and voices.Audio uses 16 kHz sampling, 256 mel bins, and 16 ms frames; windows contain up to 1,280 frames, while the score vocabulary has 248 tokens.Source: §2 · p. 1
2.1.1. Encoder
- A frozen hFT-Transformer provides pitch-wise audio features, while parallel branches produce content memory and framewise downbeat-phase estimates for the decoder.The phase branch uses a dilated TCN followed by six bidirectional Mamba layers; the content path projects frame features into 256 dimensions.Source: §2.1.1 · pp. 1–2, Figure 1 · p. 2
2.1.2. Fourier Positional Encoding of Downbeat Phase
- Downbeat phase cycles once per notated bar, and a shared 24-dimensional Fourier map encodes audio- and score-side phase using harmonics through order 12.Audio phase is represented by a posterior over 360 within-bar positions, which is used to form the audio-side Fourier positional encoding.Source: §2.1.2 · p. 2
2.1.3. Transformer Decoder
- The four-layer Transformer decoder adds score-side phase features to its inputs and combines audio content with audio-side phase features in cross-attention keys and values.Score phase is derived from the consumed score prefix relative to the current bar duration, resets at bar tokens, and is set to zero before a time signature is available.Source: §2.1.3 · p. 2
2.2. Training Objective
- Training combines teacher-forced next-token score cross-entropy with a framewise phase loss, weighted by λ = 0.1.Phase targets are quantized to 360 circular classes, smoothed across ±3 classes, and interpolated within annotated beats.Source: §2.2 · p. 2
2.3. Inference
- Whole-piece inference detects downbeat crossings, decodes signature sequences and five-bar excerpts, then assembles the excerpts into a score.Overlapping phase predictions are combined by averaging circular first moments; decoding retains complete-bar prefixes and restarts at subsequent downbeats.Source: §2.3 · pp. 2–3
3. EXPERIMENTS
Experiments evaluate the system on synthetic and human-performance datasets under pre-segmented and whole-piece protocols. The learned coordinate supports metrical structure and whole-piece transcription without external segmentation.
3.1. Data
- HumSyn and MuseSyn provide training and validation data, while Syn and ASAP are evaluated; the final Syn split has 546 training, 46 validation, and 45 test scores.Training audio varies tempo, performance style, SoundFonts, and reverberation; each score is rendered in four versions, with one selected at each epoch.Source: §3.1 · p. 3
3.4. Results from Pre-segmented Transcription
- On Syn, the coordinate significantly improves Fmeter over the model without it; on ASAP, Fmeter is slightly higher but the overall MV2H5 is almost unchanged.Both model variants outperform Piano-A2S on every component on Syn; the coordinate’s benefit for pre-segmented ASAP transcription remains inconclusive.Source: Table 1 · p. 3, §3.4 · p. 4
3.2. Training Configuration
- The 23.4M-parameter model is trained for up to 50 epochs, with validation-based checkpoint selection, early stopping, AdamW, dropout, and a scheduled learning rate.The model includes 17.9M trainable and 5.5M frozen parameters; the phase model uses a distinct learning-rate schedule.Source: §3.2 · p. 3
3.3. Metrics
- Transcription quality is measured with MV2H and OMR-NED; MV2H includes five components, with Fmeter most relevant to metrical placement, while lower OMR-NED means fewer symbol insertions and deletions.Whole-piece MV2H uses a faster single-path implementation checked against the official implementation; mean per-excerpt MV2H5 differences were 0.0059 on Syn and 0.0076 on ASAP.Source: §3.3 · pp. 3–4
3.5. Results from Self-segmented Whole-piece Transcription
- The system significantly exceeds Beat This!-segmented Piano-A2S in Fmeter and MV2H5 on both test sets, with lower OMR-NED on Syn and comparable OMR-NED on ASAP.It transcribes all recordings without external segmentation, and its Fmeter and MV2H5 advantage remains when Piano-A2S uses annotated downbeats.Source: Table 2 · p. 4, §3.5 · p. 4
- Oracle results show that corrected boundaries alone do not consistently improve metrical scores, while annotated coordinates and boundaries together give a larger improvement.Because phase and score models train on synthesized performances, generalization may depend on learning expressive timing and its relation to notated meter.Source: §3.5 · p. 4
4. CONCLUSION
The conclusion presents jointly learned downbeat phase as both a score-decoding signal and a source of bar boundaries for whole-piece transcription. It identifies expressive timing and notational conventions as directions for future work.
4. CONCLUSION
- Learned downbeat phase conditions score decoding and supplies boundaries for stitching bar segments into whole pieces without an external beat tracker.The experiments suggest the coordinate helps capture metrical structure.Source: §4 · p. 4
- Future work will study how expressive timing and notational conventions affect coordinate estimation, and explore editorial-style conditioning for expressive piano transcription.Source: §4 · p. 4
- Piano-A2S accepts 4–12 s inputs, so longer annotated segments reduce its transcription coverage.Source: §4 · p. 4