Source-linked AI summary
High-resolution Piano Transcription with Pedals by Regressing Onset and Offset Times
Qiuqiang Kong, Bochen Li, Xuchen Song, Yuan Wan, Yuxuan Wang
TL;DR
Previous piano transcription systems were limited by frame-wise onset and offset encoding, label misalignment sensitivity, and sparse sustain-pedal benchmarks. This paper regresses precise note and pedal event times and analytically recovers them, achieving higher note-onset F1 and a benchmark pedal result on MAESTRO.
Problem
Previous AMT systems limited transcription resolution to the frame hop size, were sensitive to misaligned onset and offset labels, and lacked sustain-pedal benchmarks on MAESTRO.
Method
The system regresses continuous distances from frame centers to precise piano-note onset and offset times, then analytically calculates precise note and pedal event times at inference.
Results
The system achieves a 96.72% onset F1 on MAESTRO versus 94.80% for the onsets and frames system, and a 91.86% pedal event F1.
Takeaways & Limitations
The proposed system provides high-resolution piano transcription, remains robust to misaligned onset and offset labels, and establishes a pedal-transcription benchmark on MAESTRO.
Abstract
from arXiv · showhide
Automatic music transcription (AMT) is the task of transcribing audio recordings into symbolic representations. Recently, neural network-based methods have been applied to AMT, and have achieved state-of-the-art results. However, many previous systems only detect the onset and offset of notes frame-wise, so the transcription resolution is limited to the frame hop size. There is a lack of research on using different strategies to encode onset and offset targets for training. In addition, previous AMT systems are sensitive to the misaligned onset and offset labels of audio recordings. Furthermore, there are limited researches on sustain pedal transcription on large-scale datasets. In this article, we propose a high-resolution AMT system trained by regressing precise onset and offset times of piano notes. At inference, we propose an algorithm to analytically calculate the precise onset and offset times of piano notes and pedal events. We show that our AMT system is robust to the misaligned onset and offset labels compared to previous systems. Our proposed system achieves an onset F1 of 96.72% on the MAESTRO dataset, outperforming previous onsets and frames system of 94.80%. Our system achieves a pedal onset F1 score of 91.86\%, which is the first benchmark result on the MAESTRO dataset. We have released the source code and checkpoints of our work at https://github.com/bytedance/piano_transcription.
I. INTRODUCTION
AMT converts audio into symbolic representations, while piano transcription must recover note events under challenging polyphony. Previous neural systems remain limited by frame-wise timing, target design, and sensitivity to label misalignment; this work proposes regression-based high-resolution transcription.
- AMT transcribes audio recordings into symbolic representations such as piano rolls, guitar fretboard charts, and MIDI files.
- Piano transcription recovers pitch, onset, offset, and velocity from solo recordings, but high polyphony makes the task challenging.
- Neural piano systems use audio features to predict frame-wise note presence, with onsets-and-frames models achieving state-of-the-art results.
- 32 ms frame hops limit previous systems’ time resolution, while binarized onset and offset targets provide no principled representation of attack duration.
- Previous targets can change completely when audio and labels are misaligned by one or several frames.
- The proposed system regresses distances to precise onset and offset times, analytically recovering arbitrary-resolution events and testing robustness to misaligned labels.
B. Onsets and Frames Transcription System
Onsets-and-frames transcription jointly predicts onset and frame outputs, using onset predictions as conditional information for frame-wise note estimation. Its loss combines frame and onset binary cross-entropy objectives.
- The system predicts onset and frame probabilities with separate acoustic models whose outputs have shape T × K.
- A joint loss combines frame-wise loss with onset loss to train the two prediction tasks.
- Onset predictions provide extra information for predicting frame-wise outputs, making onsets-and-frames a benchmark piano transcription system.
III. HIGH-RESOLUTION PIANO TRANSCRIPTION SYSTEM
Previous frame-based targets lose precise timing and are sensitive to label shifts. The proposed system instead regresses physically interpretable distances to nearby onset and offset times, preserving sub-frame timing information.
- Limitations of previous targets: Previous methods limit transcription resolution to the frame hop size and quantize precise onset and offset times into frames.
- Limitations of previous targets: Empirical multi-frame onset labels may not match the unknown duration of piano attacks, and shifting labels can produce completely different targets.
- Regression-based targets: The proposed target assigns each frame a value based on its time distance from the nearest precise onset or offset.
- Regression-based targets: The regression formulation predicts continuous onset and offset times rather than classifying onset and offset presence in each frame.
- Regression-based targets: The target sharpness is controlled by J, with larger J producing smoother targets and smaller J producing sharper targets.
- Regression-based targets: Regression-based onset and offset losses use binary cross-entropy while minimizing the discrepancy between predicted and target values.
B. Velocity Estimation
The system estimates piano-note velocities alongside onset, frame, and offset outputs. Velocity information is tied to note loudness and is used selectively for onset-related prediction.
- Piano-note velocity is estimated as MIDI velocity, an integer from 0 to 127 correlated with perceived loudness.
- The velocity loss is modulated by ground-truth onsets, so velocity prediction is trained only at note onsets.
- The acoustic model contains convolutional processing followed by bidirectional recurrent layers, with pooling applied along the frequency axis.
C. Entire System
The proposed system combines velocity, onset, frame-wise, and offset submodules, then converts their outputs into high-resolution piano note events. It detects local-maximum onsets and refines onset or offset times analytically from neighboring frame predictions.
- Architecture: Four submodules process the log mel spectrogram: velocity regression, onset regression, frame-wise classification, and offset regression.Predicted velocities condition onset prediction, while onset and offset outputs condition frame-wise prediction.
- Training: The training objective equally weights frame, onset, offset, and velocity losses.
- Inference: The inference algorithm outputs note events represented by piano note, onset time, offset time, and velocity.
- Inference: Precise onset or offset time is calculated from three neighboring frame predictions rather than assigning the event to the local-maximum frame time.For points A, B, and C, the calculated event time G is positioned so AG and CG are symmetric about the vertical through G.
- Inference: A note onset is detected when its onset regression exceeds a threshold and is a local maximum; velocity is scaled by Pvel(t, k) × 128.
- Inference: Offsets are detected from an offset-regression threshold and local maximum or from a frame-wise prediction below its threshold, then refined analytically.
E. Sustain Pedal Transcription
The sustain-pedal system extends the high-resolution transcription approach to pedal events using separate pedal outputs and threshold-based event detection. It models only binary pedal states and excludes half-pedal techniques.
- System design: The pedal model predicts T × 1 outputs for pedal presence, unlike the note model’s T × 88 outputs.
- Motivation: Earlier piano transcription systems often omitted sustain pedals, while prior pedal systems did not provide a benchmark on MAESTRO.
- System design: Sustain-pedal transcription is trained separately from note transcription and later combined into a unified model.Separate training improves transcription performance and reduces memory usage.
- Training: Pedal training uses continuous onset and offset regression targets together with binarized frame-wise targets.The three corresponding losses are combined in the total objective.
- Inference: Pedal onset is detected when the frame-wise prediction exceeds its onset threshold and rises relative to the previous frame.A pedal offset is triggered by either the offset regression threshold or a sufficiently low frame-wise prediction.
A. Dataset
The study uses MAESTRO V2.0.0, a large-scale paired audio–MIDI dataset of piano performances captured with high-precision MIDI systems.
- Dataset: MAESTRO V2.0.0 contains paired audio recordings and MIDI files for training and evaluating piano transcription.
- Dataset: The recordings come from the International Piano-e-Competition and were performed on Yamaha Disklaviers with high-precision MIDI capture and playback.
B. Preprocessing
Audio is converted to standardized mono, 16 kHz clips and represented with log mel spectrograms computed from 10-second segments using a 10 ms frame hop.
- Audio preparation: Stereo recordings are converted to mono and resampled to 16 kHz before being split into 10-second clips.The 16 kHz cutoff covers the frequency of piano’s highest note, C8.
- Feature extraction: A Hann-windowed short-time Fourier transform with window size 2048 produces spectrogram features, followed by 229-band log mel extraction from 30 Hz to 8000 Hz.
- Feature extraction: Figure 6 orders the visualization from spectrogram and frame-wise targets or outputs to onset and offset regression targets or outputs.
C. Model Architecture
The system uses regression targets for precise piano transcription and evaluates robustness, tolerance sensitivity, and pedal transcription on MAESTRO.
- Model architecture: The architecture predicts velocity, onset regression, frame-wise classification, and offset regression outputs from log mel spectrogram features.The acoustic models use convolutional and recurrent layers without time-axis pooling to retain temporal resolution.
- Piano transcription results: The regression-based system improves note F1 from 94.80% to 96.72% when onset, offset, frame, and velocity conditioning are combined.It also improves offset-evaluated note F1 from 79.67% to 82.47% and offset-and-velocity note F1 from 76.04% to 80.92%.
- Robustness to label misalignment: With misaligned labels, the proposed system achieves 96.39% note F1 versus 76.52% for the previous system.Labels were randomly shifted within ±50 ms; the proposed system scored 75.62% with offset and velocity evaluation.
- Tolerance evaluation: Note F1 rises from 51.28% at 10 ms offset tolerance to 91.88% at 500 ms, outperforming the previous system across tolerances.The evaluation fixes onset tolerance at 50 ms and varies offset tolerance from 10 ms to 500 ms.
- Pedal transcription: The system evaluates sustain-pedal transcription on MAESTRO alongside implemented comparison systems.The previous onsets-and-frames system lacked pedal transcription, motivating a matched implementation for comparison.
E. Error Analysis
Error analysis identifies octave-related false positives and missed notes, with degraded performance for out-of-tune pianos or low-quality recordings.
- False positives: Most false positives arise from octave errors, including harmonics recognized as notes and short-duration or repeated-note detections.Some false positives have no obvious explanation.
- False negatives: False negatives include missed higher octave notes and some ignored bass notes.
- Recording conditions: Performance degrades when the piano is out of tune or recording-device quality is low.
V. CONCLUSION
The paper proposes regression-based high-resolution transcription for piano notes and pedals, with analytical inference of precise event times. It reports strong MAESTRO results, robustness to label misalignment, and a first pedal benchmark, while noting dependence on recording quality and lack of real-time adaptation.
- Contribution: The system regresses precise onset and offset times for piano notes and pedals to provide high-resolution transcription.
- Inference: An analytical inference algorithm calculates precise onset and offset times from the model outputs.
- Results: The system achieves 96.72% onset F1, surpasses the previous 94.80% result, and reaches 91.86% pedal event F1 on MAESTRO.
- Implications: The system is robust to misaligned onset and offset labels and supports applications including GiantMIDI-Piano creation and piano performance analysis.
- Limitations: Transcription quality depends on audio-recording quality, and the system requires modification for real-time applications.