Source-linked AI summary
Enabling Factorized Piano Music Modeling and Generation with the MAESTRO Dataset
Curtis Hawthorne, Andriy Stasyuk, Adam Roberts, Ian Simon, Cheng-Zhi Anna Huang, Sander Dieleman, Erich Elsen, Jesse Engel, Douglas Eck
TL;DR
Direct neural generation of musical audio must model dependencies across many timescales, while prior approaches lacked suitable structure or interpretability. The paper uses notes as an intermediate representation in a factorized transcription, MIDI-generation, and WaveNet-synthesis pipeline, enabled by the MAESTRO dataset, and demonstrates coherent piano generation with state-of-the-art transcription results.
Problem
Musical audio generation is difficult because music has dependencies across timescales, while some prior generated representations were not interpretable or manipulable.
Method
The paper factorizes piano modeling into transcription, a relative-self-attention Music Transformer for MIDI generation, and MIDI-conditioned WaveNet synthesis, trained with MAESTRO.
Results
The system generates about one minute of coherent piano music and achieves state-of-the-art results on a piano transcription benchmark.
Takeaways & Limitations
Using notes as an intermediate representation supports interpretable manipulation, modular model substitution, and access to unlabeled audio through transcription.
Takeaways & Limitations
Extending the approach to other or multiple instruments will likely be limited by suitable training data and sufficient transcription performance.
Abstract
from arXiv · showhide
Generating musical audio directly with neural networks is notoriously difficult because it requires coherently modeling structure at many different timescales. Fortunately, most music is also highly structured and can be represented as discrete note events played on musical instruments. Herein, we show that by using notes as an intermediate representation, we can train a suite of models capable of transcribing, composing, and synthesizing audio waveforms with coherent musical structure on timescales spanning six orders of magnitude (~0.1 ms to ~100 s), a process we call Wave2Midi2Wave. This large advance in the state of the art is enabled by our release of the new MAESTRO (MIDI and Audio Edited for Synchronous TRacks and Organization) dataset, composed of over 172 hours of virtuosic piano performances captured with fine alignment (~3 ms) between note labels and audio waveforms. The networks and the dataset together present a promising approach toward creating new expressive and interpretable neural models of music.
1 INTRODUCTION
The paper factorizes expressive piano music modeling around discrete notes, using separate transcription, MIDI generation, and audio synthesis modules. This design addresses multiscale musical dependencies while enabling interpretable manipulation and training from aligned symbolic-audio data.
- Music modeling is difficult because pitch, timbre, rhythm, and song structure depend on timescales ranging from short-term to long-term.
- Prior approaches either lost coherence beyond 1 or 2 seconds, introduced fine-scale artifacts, or required labeled audio and monophonic sequences.
- An intermediate note representation is more interpretable and manipulable than directly generated audio.
- Wave2Midi2Wave splits piano modeling into transcription, autoregressive MIDI generation, and MIDI-conditioned WaveNet synthesis.
- MAESTRO supplies well-aligned audio and symbolic performances at roughly an order of magnitude larger scale than previous benchmarks.
- The factorized design supports modular replacement of performance and instrument models and modeling structure over roughly a minute of music.
2 CONTRIBUTIONS OF THIS PAPER
The paper combines factorized piano audio modeling with a substantially larger aligned piano dataset and reports state-of-the-art piano transcription results. The system generates about one minute of coherent piano music.
- The system combines transcription, language modeling, and MIDI-conditioned WaveNet synthesis for factorized musical audio modeling.
- The MAESTRO dataset contains piano performance recordings with aligned MIDI and is an order of magnitude larger than previous datasets.
- The transcription model achieves state-of-the-art results on a piano transcription benchmark.
3 DATASET
MAESTRO is a large paired piano audio–MIDI dataset built from live competition performances, with careful synchronization and publicly released splits. Its design addresses limitations in scale, alignment, and performance naturalness found in earlier datasets.
- The dataset uses Disklavier MIDI capture alongside conventionally recorded WAV audio from virtuoso piano performances.The source streams initially differed in start times, durations, and jitter, requiring alignment processing.
- MAESTRO contains over a week of paired audio and MIDI recordings from nine years of International Piano-e-Competition events.The recordings include key strike velocities, sustain pedal positions, and piece-level composer, title, and performance-year annotations.
- Audio and MIDI files are aligned with ≈3 ms accuracy and sliced into individual musical pieces.The audio is CD quality or higher, at 44.1–48 kHz, 16-bit PCM stereo.
- MAESTRO is around an order of magnitude larger than previously published paired piano audio–MIDI datasets.Earlier datasets differ in alignment quality, recording naturalness, instrument coverage, and recording environments.
- Alignment was obtained by globally minimizing distance between real-audio CQT frames and FluidSynth-synthesized MIDI.The source recordings were independent streams, making precise synchronization non-trivial.
- The provided split keeps compositions disjoint across train, validation, and test sets, targeting roughly 80/10/10 percent by duration.The authors recommend their splits for comparison but do not expect them to suit every purpose.
4 PIANO TRANSCRIPTION
The paper trains a modified Onsets and Frames piano transcription model on MAESTRO, adding architectural changes and audio augmentation. It reports state-of-the-art results on MAPS and establishes MAESTRO split baselines.
- The modified Onsets and Frames model trained on MAESTRO achieves state-of-the-art piano transcription results.The model uses several modifications selected through a coarse validation-split hyperparameter search.
- The model adds an offset-detection head whose labels cover the 32 ms following each note’s end.The offset head feeds into the frame detector but is not directly used during decoding.
- The model enlarges recurrent, convolutional, and fully connected layers and changes gradient propagation, frame loss weighting, and spectrogram frequency spacing.These modifications were part of the final transcription architecture.
- Audio augmentation was particularly important for evaluation on MAPS, where it likely improved robustness to recording-environment and piano-quality differences.Training randomized parameters for the SoX audio tool for every input sample.
- The transcription model supports creating MAESTRO-T by transcribing audio, enabling subsequent language and synthesis modeling from a larger piano-data set.The authors note that, in theory, any large unlabeled piano collection could be transcribed similarly.
5 MUSIC TRANSFORMER TRAINING
The paper trains a Music Transformer language model on symbolic piano events, using both original MAESTRO MIDI and transcriptions inferred from MAESTRO audio. Evaluation is performed on the respective validation splits.
- The generative language model uses a Transformer decoder with relative self-attention to model piano music.The architecture was selected based on prior results showing longer-term musical coherence.
- Two models are trained: one on MAESTRO MIDI and one on MIDI transcriptions inferred by Onsets and Frames, called MAESTRO-T.This compares models trained from recorded symbolic performances and automatically transcribed audio.
- Both models are trained on random crops of 2048 events with transposition and time-compression/stretching augmentation.Time-stretch factors are sampled from {0.95, 0.975, 1.0, 1.025, 1.05}.
- The models are evaluated on their respective validation splits, with validation negative log-likelihood reported for the event-based representation.The supplied passage identifies the evaluation split and metric but does not provide the table’s numerical values.
6 PIANO SYNTHESIS
The piano synthesis component uses WaveNet conditioned on symbolic note information, separating musical structure from local waveform and instrument modeling. Models were trained with unconditioned, ground-truth, or transcribed MIDI conditioning, while longer outputs could exhibit timbral shifts and initial sonic crashes.
- Conditioned synthesis: MIDI conditioning lets WaveNet focus on local instrument timbre and note interactions rather than learning long-range musical structure.The conditioning sequence supplies large-scale structure that waveform modeling alone captures less effectively.
- Conditioned synthesis: The context stack receives an 88-dimensional onset piano roll at 4 ms resolution, with values encoding piano-key strike velocities.Each frame represents keyboard onsets, and absent onsets are encoded as zero.
- Experimental variants: Three WaveNet variants were trained: unconditioned, ground-truth conditioned, and conditioned on MIDI transcribed from audio.The transcribed condition uses Onsets and Frames inference, while the other conditions use audio alone or paired ground-truth MIDI.
- Observed limitations: Heterogeneous recording conditions sometimes caused timbral shifts during longer outputs, motivating conditioning on a one-hot year vector.The year-conditioned model produced consistent timbres and ambient qualities during long outputs.
- Observed limitations: Arbitrary training windows could create a sonic crash at generated-output beginnings, so the first 2 seconds were trimmed.The trimming was applied to outputs reported in the paper and its online supplement.
7 LISTENING TESTS
The listening study compared real recordings with progressively factorized synthesis conditions using 20-second clips. Real recordings were not significantly different from ground-truth- or transcription-conditioned WaveNet samples, although the study was limited by the feasibility of testing longer clips at scale.
- Study design: The study used 20-second clips to separately assess transcription, language modeling, and synthesis within the factorized system.Each comparison relied on an additional model from the pipeline.
- Limitations: Longer listening samples would have been beneficial, but testing them at scale was not feasible.The evaluation therefore relied on 20-second clips.
- Compared systems: The full pipeline converted unlabeled piano audio into inferred MIDI, generated performances with a language model, and rendered new audio with a conditioned WaveNet.This end-to-end condition used only raw piano recordings as input.
- Study design: 640 ratings were collected, with each source participating in 128 pairwise comparisons judged by perceived resemblance to a real piano performance.Participants used a Likert scale, and Figure 2 reports how often each source won.
- Results: Real recordings did not differ significantly from WaveNet Ground/Test or WaveNet Transcribed/Test samples, with p > 0.01/10.A Kruskal-Wallis test found at least one significant difference among models: χ2(2) = 67.63, p < 0.001.
8 CONCLUSION
The paper concludes that Wave2Midi2Wave provides a factorized piano modeling system enabled by MAESTRO, with transcription generalizing beyond the dataset. Future extensions to other or multiple instruments depend on suitable data and transcription quality.
- Conclusion: Wave2Midi2Wave combines factorized models enabled by the MAESTRO dataset for piano music modeling.The authors demonstrated the system's capabilities on the same dataset.
- Future work: Extending the approach to other or multiple instruments is identified as future work, with suitable datasets and transcription performance likely to be limiting factors.The stated scope boundary concerns both training data and transcription quality.
- Conclusion: The transcription model generalized beyond MAESTRO recordings on the separate MAPS dataset.The authors cite the MAPS benchmark results as evidence of effective generalization.
- Conclusion: The dataset, including MIDI, audio, metadata, and split configurations, was publicly released under a Creative Commons Attribution Non-Commercial Share-Alike 4.0 license.The release supports reuse of the paper's aligned piano resources.
APPENDIX
The appendix describes a multi-stage procedure for aligning long MIDI recordings with audio, validating the alignment, and segmenting performances into composition-disjoint examples. It combines CQT-based matching, constrained DTW refinement, silence-based segmentation, trimming, and padding.
- Alignment: The alignment process synthesizes MIDI audio and minimizes an audio-based difference metric to find shifts matching each recording pair.The approach relies on recognizing shared score content despite differences in instruments or recording equipment.
- Alignment: The initial CQT alignment used approximately 90 ms hops as a speed–accuracy trade-off, while later refinement used approximately 3 ms resolution.The refinement stage extracted CQTs with hop length 64 at 22,050 Hz.
- Alignment: Normalized CQT windows were compared with mean squared error to align shorter audio files against long synthesized-MIDI recordings.Minimum error selected the best alignment before proceeding through the remaining MIDI sequence.
- Validation: Alignment quality was empirically supported by approximately 2–2.5x metric separation between aligned pairs and completely different audio segments.After parameter tuning, spot checks of the highest-MSE pairs confirmed proper alignment before segmentation.
- Segmentation: Pairs were segmented at long MIDI silences, with non-musical note clusters trimmed and one second of padding added at both ends.The segmentation enabled train, validation, and test splits disjoint on musical compositions.
- Alignment: Dynamic Time Warping corrected residual audio–MIDI jitter, using a custom C++ implementation for long sequences.The refined procedure restricted cosine-distance calculations to a 2.5-second Sakoe-Chiba band.