PaperlayerView exact revision on arXiv

Source-linked AI summary

Phase-Based Onset Enhancement for Piano Transcription in Ensemble Settings

Hyemi Kim, Jiyun Park, Juhan Nam

arXiv:2610.29222v1cs.SD

TL;DR

Piano transcription performance degrades in ensemble settings, where loud sustained instruments can obscure piano notes in magnitude representations. This paper combines phase information with magnitude features and evaluates the approach on ensemble datasets; it reports improved performance over its baseline in that setting.

  • Problem

    Piano transcription methods that perform well on solo recordings degrade in ensemble settings, where loud sustained sounds can make piano onsets less distinguishable in magnitude representations.

    Source: §1 · p. 1

  • Method

    The proposed model incorporates phase information alongside magnitude features and evaluates on MAESTRO ensemble recordings and the newly constructed PCD-K dataset.

    Source: §1 · p. 2

  • Results

    The proposed method improves performance in ensemble settings and outperforms the baseline model trained with the same additional Symphony dataset.

    Source: §7 · p. 6

  • Takeaways & Limitations

    The paper identifies phase information as useful for improving piano transcription in classical ensemble settings such as trios and concertos.

    Source: §7 · p. 6

    Velocity evaluation is ill-defined in polyphonic settings, and dataset-dependent velocity variation may not be captured by the metric.

    Source: §6.3 · p. 6

Abstract

from arXiv · show

Piano transcription models have achieved very high performance in recent state-of-the-art systems, but their performance tends to degrade in ensemble settings where the piano is played alongside other instruments. While some approaches attempt to address this issue by incorporating multi-instrumental datasets, the problem remains challenging. In particular, when other instruments produce high-energy sustained tones while relatively low-energy piano notes occur simultaneously, the magnitude components of their frequency representations overlap, making it difficult to accurately distinguish piano notes. To address this issue, we propose an improved piano transcription model that jointly leverages magnitude and phase information, enabling more robust piano transcription performance in ensemble settings. We evaluate the proposed model on two piano ensemble datasets with precise MIDI note annotations, which we obtained by revisiting and extending existing datasets. Notably, we constructed a new piano concerto dataset, PCD-K, by playing a piano part on a Disklavier and recording the audio and MIDI pairs with pre-recorded orchestral tracks. Experimental results demonstrate that phase information is highly effective in improving performance, outperforming magnitude-based approaches in ensemble settings.

1. INTRODUCTION

Piano transcription performance degrades in ensemble settings, where loud sustained instruments can mask weaker piano onsets in magnitude-based representations. The paper combines phase with magnitude features and evaluates the approach using ensemble datasets with piano MIDI annotations.

  • Ensemble settings cause piano transcription performance to degrade, despite high performance on solo piano recordings.The challenge includes accompaniment, piano trios, and concertos with a full orchestra.Source: §1 · p. 1
  • Loud sustained orchestral tones can obscure low-energy piano onsets in magnitude representations, especially at shared pitches or overlapping harmonics.The masking is more severe when instruments share a pitch or have overlapping harmonics.Source: §1 · p. 1
  • Abrupt phase variations induced by piano onsets can provide a cue when magnitude-based representations fail to capture weak notes.The proposed explanation contrasts stable phase during sustained orchestral tones with abrupt phase variation at piano onsets in the same frequency bands.Source: §1 · pp. 1–2
  • The proposed model combines phase variations with magnitude features, using Transkun as its transcription backbone and Symphony data augmentation.Transkun is a semi-CRF-based piano transcription framework that also supports pedal transcription.Source: §1 · p. 2
  • The evaluation uses ensemble recordings identified in MAESTRO and the new PCD-K dataset, recorded with pianists on a Disklavier to capture audio and MIDI.PCD-K extends the Piano Concerto Dataset, whose audio–MIDI-aligned annotations were not sufficiently accurate for evaluating transcription models.Source: §1 · p. 2
  • The paper compares note and sustain-pedal transcription against conventional approaches on datasets for classical piano trios and concertos.The authors identify and construct evaluation datasets for both ensemble settings.Source: §1 · p. 2

2. RELATED WORKS

Prior piano transcription work spans frame-based prediction and methods that model note structures or temporal dependencies more directly. The reviewed methods use magnitude-based representations, while available datasets separate precise solo labels from less accurate ensemble annotations.

  • Frame-based approaches estimate onset, offset, and pitch-related signals before post-processing them into note events.Onsets and Frames separated onset and frame predictions; later work added offset regression, adversarial learning, note-state modeling, autoregressive connections, harmonic priors, or multitask feature modulation.Source: §2 · p. 2
  • Other approaches model transcription as audio-to-token translation, hierarchical time–frequency attention, language-model note generation, or explicit note intervals.These methods also rely on magnitude-based mel-spectrogram or CQT features and do not explicitly use phase information.Source: §2 · p. 2
  • Available datasets pair precise labels with solo performances, while ensemble datasets use score-aligned annotations that are less accurate for transcription.The paper identifies a need for piano ensemble data with precise transcription labels.Source: §2 · p. 2

3. PROPOSED METHOD

The method builds on Transkun’s interval-based semi-CRF transcription and augments its interval scores with phase-derived onset saliency. This enhancement is applied to piano-key scores, not pedal-control scores.

3.1 Revisiting Transkun

  • Transkun formulates transcription as interval-based structured prediction, scoring candidate events within a probabilistic model over non-overlapping event sequences.Its encoder-only transformer uses scaled inner-product interval scoring, and its note-level formulation avoids post-processing.Source: §3.1 · p. 2

3.2 Transkun-Phase

  • The enhancement is applied only to the 88 piano-key score maps, because note onsets create abrupt phase discontinuities while pedal events preserve phase continuity.The model estimates onset probability for each frame and piano pitch with an onset-prediction head.Source: §3.2 · p. 3, Figure 2 · p. 3
  • Phase-derived onset saliency is added to mel-based interval scores, boosting intervals that begin at likely onset frames.The phase-based score is shared across intervals with the same start frame, and λphase controls its contribution.Source: §3.2 · pp. 2–3
  • Training combines the semi-CRF conditional-likelihood loss and event-attribute losses with a binary cross-entropy loss for predicted onset probabilities.Attribute losses cover velocity and refined onset and offset predictions.Source: §3.2 · p. 3

4. EVALUATION DATASETS

The evaluation uses six MAESTRO v2 piano-trio recordings and introduces PCD-K, a concerto dataset with newly recorded piano audio and performance MIDI paired with orchestral tracks.

4.1 MAESTRO v2 Trio

  • The MAESTRO v2 Trio subset contains six piano, violin, and cello recordings with piano note and pedal annotations, totaling about 2 hours and 33 minutes.Source: §4.1 · p. 3

4.2 PCD-K: Piano Concerto Dataset for Piano Transcription

  • PCD-K replaces PCD’s alignment-derived piano MIDI, considered insufficiently accurate for transcription, with recordings made by re-performing the piano parts on a Disklavier while capturing audio and MIDI together.Performers synchronized to the orchestral tracks, with manual timing adjustments when needed.Source: §4.2 · pp. 3–4
  • PCD-K comprises 70 excerpts, including 59 single-version PCD excerpts and 11 new excerpts, with non-reverb and reverberant versions to reflect acoustic variability.The piano pitch histogram is shown in Figure 3.Source: §4.2 · p. 4, Figure 3 · p. 4

5. EXPERIMENTAL SETTINGS

The experimental setup augments piano recordings with orchestral material, specifies a magnitude-and-phase input and training procedure, and compares the proposed model with pretrained piano and multi-instrument systems.

5.1 Symphony: Additional Training Dataset

  • The Symphony dataset supplies piano-free orchestral recordings for training mixtures, while the MAESTRO piano track is randomly mixed with orchestral tracks at varied gains.Symphony covers strings, woodwinds, brass, and percussion; it contains about 25 hours across 13 composers, split 9:1 for training and validation.Source: §5.1 · p. 4

5.2 Model Specification

  • The model uses STFT magnitude and phase alongside log-mel inputs, and follows Transkun training with AdaBelief, cosine annealing, warm-up, and gradient clipping.Training runs for 500K iterations with batch size 8; the phase-based onset weight is set to 100 and γ to 1.Source: §5.2 · p. 4

5.3 Model Comparison

  • Comparisons include Transkun, Transkun_Aug, Kong, and YourMT3+, using frame- and note-based metrics with specified onset, offset, and velocity tolerances.Kong is excluded from the MAESTRO v2 Trio evaluation because it had seen five of the six pieces during training; Transkun’s MAESTRO-only model serves as a lower bound on mixtures and upper bound on isolated piano.Source: §5.3 · pp. 4–5

6. EXPERIMENTAL RESULTS

The proposed method outperforms the Transkun baseline in overall F1-scores and recall across both ensemble datasets. Results and ablations favor magnitude combined with first-order phase information, though velocity evaluation is an exception.

6.1 Evaluation on MAESTRO v2 Trio

  • The lower-bound Transkun model has low precision but high note-onset recall and many false positives; additional training data improves precision and onset F1.Transkun_Aug consistently outperforms YourMT3+, which performs substantially worse on note onset with offset and velocity.Source: §6.1 · p. 5
  • On MAESTRO v2 Trio, the proposed method scores 94.7%, 83.9%, and 82.8% F1 for note onset, note onset with offset, and note onset with offset and velocity.The Transkun baseline scores 93.6%, 82.2%, and 80.8% on those respective metrics.Source: §6.1 · p. 5

6.2 Evaluation on PCD-K

  • Among pretrained PCD-K models, Transkun_Aug performs best overall, especially on note onset with offset and frame-level metrics, while YourMT3+ leads slightly on note onset.Kong follows Transkun_Aug on note-onset-with-offset metrics.Source: §6.2 · p. 5
  • Across both datasets, the proposed method achieves higher overall F1-scores than Transkun, with significant recall improvements; Figure 4 highlights recovered true positives.The figure marks recovered true positives in sky blue.Source: Figure 4 · p. 6, §6.2 · p. 6
  • On PCD-K, the proposed method reports F1-scores of 90.5%, 71.0%, and 55.5%, improving on the Transkun model.The corresponding Transkun scores are 87.9%, 68.1%, and 52.8%.Source: §6.2 · pp. 5–6
  • On PCD-K, Transkun_Aug also leads frame-level and sustain-pedal prediction, attributed to environmental-sound augmentation improving robustness under out-of-distribution conditions.The passage relates this benefit to subtle pedal-signal energy variations, including decay and reverberation.Source: §6.2 · p. 6

6.3 Ablation study

  • The magnitude-plus-first-order-phase variant performs best on MAESTRO v2 Trio, while magnitude-only drops; first-order phase remains best overall on PCD-K.The second-order variant is comparable on MAESTRO v2 Trio; on PCD-K, magnitude-only shows a moderate note-onset and clear pedal-onset F1 drop.Source: Table 5 · p. 6, §6.3 · p. 6
  • Magnitude-only is a slight exception for note-onset F1 with offset and velocity, where it scores higher than the phase-based variant.The authors note that velocity evaluation is inherently ill-defined in polyphonic settings, and values can vary across recordings.Source: §6.3 · p. 6

7. CONCLUSIONS

The paper introduces phase information to improve piano transcription in ensemble settings and prepares two evaluation datasets with accurate piano annotations. The proposed method outperforms a baseline trained with the same additional Symphony dataset, with notable recall gains.

7. CONCLUSIONS

  • The proposed phase-based method improves piano transcription in ensemble settings, particularly classical piano trios and concertos, and outperforms the same-Symphony-data baseline on note and pedal F1-scores with notable recall gains.The paper prepares a MAESTRO v2 piano-trio subset and PCD-K, created by re-performing and re-recording PCD piano tracks with MIDI capture; it identifies masked-note onset saliency as the mechanism and suggests broader transcription and score-following applications.Source: §7 · p. 6
Loading 2610.29222v1…