Source-linked AI summary
Multi-Task Multi-Frame Visual Piano Transcription
Yonghyun Kim, Hoyeol Sohn, Juhan Nam, Alexander Lerch
TL;DR
Physical key release remains difficult to recover from audio, while visual piano transcription has largely not addressed complete note attributes. V2N jointly predicts onset, offset, key hold, and velocity across video frames, substantially improving physical key-release accuracy and achieving state-of-the-art results on PianoVAM and R3.
Problem
Audio offsets are confounded by sustain-pedal reverberation, while visual piano transcription remains underexplored despite directly observing physical key states.
Method
V2N jointly trains onset, offset, key-hold, and velocity heads with per-frame supervision over 1 s of video.
Results
V2N improves physical key-release accuracy across PianoVAM and R3, with PianoVAM +Off F1 reaching 89.5 versus PPAN’s 45.9 at 50 ms.
Takeaways & Limitations
V2N provides complete video-only MIDI transcription across pitch, onset, offset, and velocity, establishing a broader scope for visual piano transcription.
Takeaways & Limitations
Cross-dataset transfer fails almost completely across camera geometries, and V2N does not estimate sustain-pedal control changes.
Abstract
from arXiv · showhide
Audio-based piano transcription performs well on onset, pitch, and velocity, but the sustain pedal lets sound persist long after key release, so audio systems predict pedal-extended offsets rather than physical key release. Yet existing Visual Piano Transcription (VPT) systems focus on onset detection from short video windows, offset accuracy lags onset by a wide margin, and note-level velocity has not been reported. To address these gaps, we present V2N (Video to Notes), the first complete VPT system: a shared temporal backbone feeds task-specific heads for onset, offset, key hold, and velocity, jointly trained with per-frame supervision rather than only at the window center. Ablations show that multi-task supervision enables offset and velocity prediction while improving onset accuracy; longer temporal context yields further improvements. V2N sets new state-of-the-art results on PianoVAM and R3.
1. INTRODUCTION
V2N addresses key limitations in Visual Piano Transcription by predicting onset, offset, key hold, and velocity from longer video context with per-frame multi-task supervision. It achieves complete video-only MIDI transcription and state-of-the-art results while exposing remaining cross-dataset transfer limitations.
- Motivation: Audio transcription confounds physical key release with sustain-pedal persistence, whereas VPT directly observes key state regardless of pedal condition.VPT can also operate when audio is ambiguous, degraded, or absent.
- Limitations: Existing VPT methods use at most 0.2 s of context, have substantially lower offset than onset accuracy, or omit note-level velocity.
- V2N: V2N jointly trains onset, offset, key-hold, and velocity heads over 1 s of video with supervision at every frame.This is presented as the first complete VPT system.
- Results: V2N matches Li et al. on PianoVAM Onset, surpasses prior VPT on R3 Onset, improves physical key-release prediction, and reports note-level velocity F1 for the first time.The system achieves state-of-the-art results on PianoVAM and R3.
- Ablations: Ablations show that multi-frame Conformer-style modeling improves onset, multi-task heads improve onset and offset, and removing auxiliary heads collapses offset F1.Offset-guided decoding terminates notes at offset-head peaks, using key hold as a fallback.
- Limitations: Cross-dataset transfer remains problematic: V2N and prior VPT systems fail similarly, highlighting limitations of fixed-geometry preprocessing.
2. RELATED WORK
Related work established multi-task audio transcription and progressively stronger visual piano transcription systems, but existing VPT methods remained limited in temporal scope and note-level velocity or offset evaluation. Audio methods extend offsets through sustain, whereas visual key-state modeling distinguishes physical release from pedal-extended audio activity.
- Audio Piano Transcription: Audio piano transcription evolved from Onsets and Frames’ multi-task prediction to Transformer, high-resolution onset/offset, and neural semi-CRF systems.Onsets and Frames jointly predicts onsets, sustained audio activity, and velocity; later work refines the paradigm with stronger architectures and decoding.
- Cross-Modal Difference: Audio transcription extends note offsets through sustain, while video tracks the physical key state under the sustain pedal.The physical key state can decouple from audio activity when pedal sustain prolongs sound after release.
- Visual Piano Transcription: Existing VPT methods progressed from background subtraction to end-to-end ResNet, multi-scale attention, and Vision Transformer approaches.S2S uses 5-frame windows, V2R predicts a center-frame pressed-key roll, and PPAN applies a Vision Transformer to 6-frame inputs.
- Remaining Gaps: Velocity has been explored with optical-flow CNNs without note-level evaluation, while a video-audio system derives offsets acoustically and does not report note-level velocity.These limitations leave note-level velocity and visually grounded offset prediction insufficiently addressed in prior work.
3. METHODOLOGY
V2N models piano transcription as four per-frame visual prediction tasks—onset, physical offset, key hold, and velocity—using a shared temporal backbone with task-specific heads. Its video architecture combines frame-wise visual features, temporal convolutions, and dense key-hold supervision to predict note events across time.
- Visual prediction targets: V2N predicts onset, physical key release, key hold, and velocity for every video frame and piano key.Onset and offset represent key mechanics, while key hold spans [onset, offset); velocity is the fourth visual target.
- Event decoding: NoteOn and NoteOff events are decoded from peaks in onset and offset predictions, while key-hold labels cover the full held interval.Repeated same-key onsets form independent note pairs, and overlapping key-hold labels merge by element-wise maximum.
- Architecture: The architecture combines a per-frame visual feature extractor, a shared temporal backbone, and task-specific prediction heads.Four parallel BiLSTM-to-linear heads produce onset, offset, key-hold, and velocity outputs.
- Feature extraction and temporal modeling: A ResNet-18 visual extractor processes jointly aggregated 5-frame windows, while three kernel-31 Conformer ConvModule blocks provide temporal context across the 1-second input.The extractor uses grayscale frames, a learned slope prior, 5-frame aggregation, global average pooling, and projection to the backbone dimension.
- Training supervision: Key hold supplies dense supervision over [onset, offset), complementing sparse onset and offset event targets and strengthening temporal representation learning.Onset and offset labels use soft triangular kernels, while key-hold labels are binary throughout the physical hold interval.
4. EXPERIMENTAL SETUP
Experiments use PianoVAM and R3 with defined train/validation/test splits and dataset-specific preprocessing. V2N and video-only baselines are evaluated through frame-level decoding into MIDI events under explicit onset, offset, velocity, and tolerance conventions.
- Datasets: PianoVAM provides 107 top-view videos with synchronized MIDI; experiments train on 81 recordings, validate on 9, and test on 9 recordings containing 42,241 notes over 1.63 hours.Eight recordings assigned to special(blurry) and special(4hands) splits are excluded.
- Datasets: R3 contains 31 hours from 62 sessions by two professional pianists and is split into R3s and R3x with separate train/test counts.R3 videos are downsampled primarily from 60 fps to 25 fps and transformed to a common 800×144 format.
- Baselines: S2S, V2R, Li et al., and PPAN are retrained as video-only baselines using author defaults or official implementations, with CNN-based reproductions for S2S and V2R.Li et al. is evaluated without audio input.
- Decoding: Frame-level onset, offset, key hold, and velocity predictions are decoded into MIDI, beginning notes at onset peaks above τon=0.5 and ending them at offset or key-hold criteria.The note ends at the earlier applicable offset peak or key-hold drop when both are available.
- Metrics: Evaluation reports frame multipitch and note-level onset, offset, velocity, and combined metrics at 50 ms and 100 ms onset tolerances, with offset tolerance matching onset tolerance.Predicted timestamps are quantized to the 25 fps grid, while ground-truth MIDI timestamps remain continuous.
5. RESULTS
V2N outperforms prior systems on offset-dependent metrics while maintaining or improving onset accuracy on PianoVAM and R3, and qualitatively reconstructs note durations and dynamics. Ablations attribute these gains to complementary multi-task heads, multi-frame supervision, sequence modeling, and longer context, while cross-dataset transfer collapses because keyboard geometry differs.
- Main results: On PianoVAM, V2N matches Li et al. on Onset within 0.5 %p at both tolerances and nearly doubles +Off F1 over PPAN at 50 ms.The +Off F1 increases from 45.9 to 89.5, while +Off+Vel rises from 29.7 to 78.3.
- Main results: On R3, V2N surpasses prior VPT on Onset at both splits and tolerances, while onset-centric baselines fail on +Off despite competitive onset accuracy.The result supports dedicated offset supervision rather than relying on key-hold modeling alone.
- Qualitative analysis: V2N reconstructs both note durations and dynamic contours, whereas Li et al.’s offsets collapse to near-zero duration and PPAN predicts constant velocity.Figure 3 presents this comparison on a representative 10 s PianoVAM excerpt.
- Multi-task heads: Velocity supervision and decoding raise +Vel F1 to 80.6 at 50 ms, while onset and offset heads provide additional supervision and decoding cues.Onset-guided decoding improves Onset F1 from 93.3 to 94.3, and offset-guided decoding yields the best +Off and +Off+Vel scores.
- Multi-frame context: Supervising all five frames yields +1.3 %p Onset and +4.5 %p +Off, while sequence modeling adds +2.5 %p Onset and +5.7 %p +Off.The multi-frame loss adds supervised signal without inference cost, and the Conformer resolves pressed-versus-held ambiguity.
- Context and transfer: Extending context from 0.2 s to 1.0 s adds +0.4/+0.8 %p (Onset/+Off) on PianoVAM but reaches 91.7% R3s Onset F1 and 88.9% +Off on R3.Cross-dataset Onset F1 collapses near zero because the fixed 800×144 warp does not standardize keyboard geometry.
6. DISCUSSION
The discussion identifies substantial R3 synchronization errors and two main limitations: poor cross-dataset transfer across camera geometries and missing sustain-pedal control estimation. Excluding synchronization-affected files substantially improves V2N’s reported R3 metrics.
- R3 synchronization issues: Video–MIDI offsets exceeding 200 ms occur in 70 of 895 R3 files, with MIDI systematically preceding video.The offsets were identified by cross-correlating MIDI onset trains with video audio energy at 10 ms resolution over a ±3 s search.
- Limitations: Cross-dataset transfer fails almost completely because learned pixel-to-key mappings do not generalize across camera geometries, and V2N omits sustain-pedal control changes.Proposed future work includes geometry-invariant keyboard localization, joint synchronization alignment, sustain-pedal estimation, and audio-visual fusion.
7. CONCLUSION
V2N is a multi-task, multi-frame visual piano transcription system that jointly predicts onsets, offsets, key holds, and velocity. It improves physical key-release accuracy over prior video baselines and is the first VPT system to report note-level velocity.
- Conclusion: V2N jointly predicts onsets, offsets, key hold states, and velocity through multi-task, multi-frame modeling.This design targets complete note-level transcription.
- Conclusion: V2N substantially improves physical key-release accuracy over prior video baselines.Earlier baselines either collapse offsets to near-zero duration or infer them post-hoc from key hold activations.
- Conclusion: V2N is the first visual piano transcription system to report note-level velocity.The system therefore extends reported VPT outputs beyond onset-focused transcription.