Source-linked AI summary
Onsets and Frames: Dual-Objective Piano Transcription
Curtis Hawthorne, Erich Elsen, Jialin Song, Adam Roberts, Ian Simon, Colin Raffel, Jesse Engel, Sageev Oore, Douglas Eck
TL;DR
Polyphonic piano transcription remains difficult because overlapping notes and changing energy challenge accurate note identification. The paper jointly predicts onsets and frames, conditions frame predictions on onset detections, and extends the model with relative velocity prediction. It reports large improvements in transcription quality, including over a 100% relative improvement in note F1 score with offsets on MAPS, while velocity-aware outputs sound more natural.
Problem
Polyphonic piano transcription is difficult even for humans, and existing evaluation must better reflect perceptually relevant timing, note durations, and dynamics.
Method
A jointly trained convolutional and recurrent network predicts pitch onsets and frames, conditions framewise predictions on onsets, and adds a separate stack for relative velocity prediction.
Results
Over a 100% relative improvement in note F1 score with offsets is reported on MAPS, while onset use improves note-with-offset quality and velocity prediction produces more natural-sounding transcriptions.
Takeaways & Limitations
Onset information is important for high-quality note-with-offset transcription, and relative velocity prediction helps retain expressive dynamics in piano transcriptions.
Takeaways & Limitations
Evaluation on restricted subsets or test-piano training data can raise scores but limits generalizability, motivating larger and more representative datasets.
Abstract
from arXiv · showhide
We advance the state of the art in polyphonic piano music transcription by using a deep convolutional and recurrent neural network which is trained to jointly predict onsets and frames. Our model predicts pitch onset events and then uses those predictions to condition framewise pitch predictions. During inference, we restrict the predictions from the framewise detector by not allowing a new note to start unless the onset detector also agrees that an onset for that pitch is present in the frame. We focus on improving onsets and offsets together instead of either in isolation as we believe this correlates better with human musical perception. Our approach results in over a 100% relative improvement in note F1 score (with offsets) on the MAPS dataset. Furthermore, we extend the model to predict relative velocities of normalized audio which results in more natural-sounding transcriptions.
1. INTRODUCTION
The paper addresses difficult polyphonic piano transcription by explicitly modeling note onsets and conditioning framewise detection on them. It also adds relative velocity prediction to preserve expressive performance information and produce more natural-sounding transcriptions.
- Motivation: Piano transcription is difficult even for humans because polyphony and decaying note energy complicate accurate note identification.Models must adapt to notes with varying amplitude and harmonics.
- Onset-conditioned transcription: Explicit onset modeling lets framewise note detection use the beginning of each note, which is typically easier to identify and perceptually significant.Piano onsets are near peak amplitude, percussive, and spectrally distinctive.
- Onset-conditioned transcription: The onset-conditioned model achieves state-of-the-art performance across frame, note, and note-with-offset transcription metrics.The authors describe the improvement as large and attribute it to conditioning on onset information.
- Expressive transcription: Relative velocity prediction adds expressive dynamics to transcriptions and results in more natural-sounding outputs.Velocity reflects how loudly a piano note sounds and is directly related to key-depression speed.
2. DATASET AND METRICS
The evaluation uses MAPS with synthesized pieces for training and Disklavier recordings for testing, while separating music pieces across splits. It combines frame- and note-level metrics to assess transcription quality, including note durations and perceptually relevant performance information.
- Dataset: MAPS provides audio and annotations for isolated notes, chords, and complete piano pieces, including synthesized and Disklavier recordings.The training split uses synthesized pieces, while the test split uses Disklavier performances.
- Dataset: The train and test sets exclude overlapping music pieces, and withholding Disklavier recordings, notes, and chords from training better reflects real-world testing.Testing on Disklavier recordings also targets transcription of real piano performances.
- Metrics: Evaluation reports frame- and note-level precision, recall, and F1, with note-with-offset scores requiring onset agreement and duration agreement.Offsets must produce durations within 20% of ground truth or within 50ms, whichever is greater.
- Metrics: The evaluation seeks measures aligned with perceptually relevant performance information because spurious or prematurely terminated notes can inflate frame scores.Onsets matter, but onset-only transcriptions cannot determine appropriate note durations without heuristics.
3. MODEL CONFIGURATION
The model jointly detects piano note onsets and framewise activations, using onset predictions to condition frame detection and constrain note starts during inference. Its training emphasizes early note frames and extends transcription with velocity prediction and velocity-aware evaluation.
- Joint onset-frame modeling: The frame detector receives onset-detector outputs as additional input, and inference permits a new note only when onset detection agrees.The onset detector uses a 0.5 threshold during inference.
- Network architecture: The network uses convolutional acoustic models with bidirectional LSTMs and 88-output sigmoid layers for onset and frame prediction.The input is a mel-scaled log-amplitude spectrogram, and the entire input sequence is presented to the network.
- Joint onset-frame modeling: Onset and frame detectors are trained with two cross-entropy losses, one for onset probabilities and one for framewise note activations.Onset labels use a 32 ms onset length, while frame labels indicate pitch activity in each frame.
- Loss design: Higher weights on early note frames incentivize accurate note beginnings, reflecting the perceptual significance of onset frames as piano energy decays.The weighted frame loss is built from a raw cross-entropy frame loss, with c = 5.0 selected by coarse hyperparameter search.
- Velocity estimation: A separate convolutional stack predicts normalized onset velocities, which are clipped and mapped to MIDI velocities for audio rendering.The velocity stack is trained independently from the onset and frame stacks; the final mapping was chosen because it produced pleasing audio renderings.
- Velocity evaluation: Velocity-aware evaluation rescales estimated velocities by fitted global scale and offset before accepting pitch-and-timing matches with velocity error below τ = 0.1.The regression minimizes squared differences between matched reference and estimated velocities.
4. EXPERIMENTS
The experiments compare the onsets-and-frames model with reimplemented baselines and commercial software on MAPS, showing strong transcription quality and fast inference. An example illustrates how onset-based filtering removes spurious frame activations.
- The evaluation compares the proposed model with reimplemented models from prior work and Melodyne, while AnthemScore is excluded because its quantized MusicXML output prevents accurate comparison.
- The onsets-and-frames model produces the best frame-level scores and note-based scores that include offsets, while also improving note-based onset scores.
- Onset-based inference filtering removes short-lived notes and brief reactivations when no corresponding onset is detected.
- 70× faster than real time, the network performs rapid inference on a Tesla K40c despite not being optimized for speed.
5. ABLATION STUDY
The ablation study tests the contributions of onset modeling, inference conditioning, recurrent design, training choices, and input representations. It also exposes dataset and alignment limitations relevant to interpreting the results.
- The ablation study evaluates removing onset components, changing bidirectional to unidirectional RNNs, altering training connections and weighting, and replacing the input representation.
- Not using onset information during inference causes an 18% relative decrease in note onset score and a 31% decrease in note-with-offset score, despite a slight frame-score increase.
- Removing the onset detector produces the worst performance on all metrics, although it still outperforms the baseline model from [13].
- Other tested architectures and augmentation strategies produce no improvement, suggesting a need for larger real-piano datasets with fully accurate label alignments.
6. NEED FOR MORE DATA, MORE RIGOROUS EVALUATION
MAPS is useful but limited in scale and representativeness, while common evaluation shortcuts can inflate scores and existing metrics may miss timing, offset, and velocity errors. The authors therefore argue for larger datasets and stricter, more musically relevant evaluation.
- Dataset limitations: The MAPS dataset has only 60 recordings, motivating a larger dataset for measuring transcription quality.The authors nevertheless recommend making full use of the available data until a larger resource exists.
- Evaluation rigor: Restricting evaluation to close recordings and short clips raises note F1 from 82.29 to 84.34 and then 86.38.These settings are described as unrepresentative of real-world transcription tasks.
- Dataset limitations: The Disklavier appears to omit some low-velocity notes, and the model performs particularly poorly below ground-truth velocity 30.The authors state that more analysis is needed to determine how frequently these missed notes occur.
- Evaluation rigor: Note-with-offset and velocity is identified as the best currently available comparison metric because onset and frame scores alone may miss musically relevant errors.The authors support this position with audio examples illustrating shortcomings of existing metrics.
- Evaluation rigor: A 50ms onset tolerance can allow excessive timing jitter, while tighter timing requirements become difficult because available datasets have limited timing accuracy.At 24ms resolution, the reported scores were Frame 76.87, Note F1 82.54, and Note-with-offset 49.99, with more extraneous notes in examples.
7. CONCLUSION AND FUTURE WORK
The authors conclude that jointly modeling onsets and frames substantially improves polyphonic piano transcription and transfers across different train and test distributions. Their examples suggest strong musical coverage, while future work targets larger datasets, language models, and richer audio representations.
- Conclusion: The jointly trained onsets-and-frames model yields significant improvements for polyphonic piano transcription by using onset information.The conclusion also states that the model transfers well between disparate training and test distributions.
- Future work: Future work includes creating a larger, more representative dataset, combining the acoustic model with a language model, and moving beyond traditional spectrogram representations.The proposed dataset expansion covers piano recording environments and music genres for training and evaluation.
- Conclusion: The system captures harmony, melody, rhythm, and dynamics in examples, with audible differences compared with other systems.The authors also report successful transcription of unrelated Musopen recordings and a harpsichord recording.