Source-linked AI summary
MIDI-VAE: Modeling Dynamics and Instrumentation of Music with Applications to Style Transfer
Gino Brunner, Andres Konrad, Yuyi Wang, Roger Wattenhofer
TL;DR
MIDI-VAE addresses the limited progress of neural style transfer for complete musical compositions. It uses parallel VAEs with a shared latent space and style classifier to transform pitches, dynamics, and instrumentation in polyphonic multitrack music. The authors report successful style transfer, smooth harmonic interpolations, medleys, and song mixtures.
Problem
Neural style transfer has progressed more slowly for sequential music, with no prior way to know how complete songs would sound in another musical style.
Method
MIDI-VAE uses parallel VAEs with a shared latent space and style classifier, representing pitch, velocity, and instrument information from multitrack MIDI.
Results
MIDI-VAE performs style transfer by changing pitches, dynamics, and instruments, and produces smooth interpolations, medleys, and harmonic mixtures of songs.
Takeaways & Limitations
The model provides a symbolic-music framework for transferring style across complete compositions while preserving polyphonic, multitrack musical structure.
Takeaways & Limitations
Increasing β can improve latent disentanglement but negatively affects reconstruction performance.
Abstract
from arXiv · showhide
We introduce MIDI-VAE, a neural network model based on Variational Autoencoders that is capable of handling polyphonic music with multiple instrument tracks, as well as modeling the dynamics of music by incorporating note durations and velocities. We show that MIDI-VAE can perform style transfer on symbolic music by automatically changing pitches, dynamics and instruments of a music piece from, e.g., a Classical to a Jazz style. We evaluate the efficacy of the style transfer by training separate style validation classifiers. Our model can also interpolate between short pieces of music, produce medleys and create mixtures of entire songs. The interpolations smoothly change pitches, dynamics and instrumentation to create a harmonic bridge between two music pieces. To the best of our knowledge, this work represents the first successful attempt at applying neural style transfer to complete musical compositions.
1. INTRODUCTION
MIDI-VAE addresses the slower progress of neural style transfer for sequential music by changing style in complete musical compositions. Its shared-latent VAE architecture transfers pitches, dynamics, and instrumentation while modeling harmonic polyphonic music.
- The work targets a gap in sequential-data style transfer, where progress has been slower than in visual style transfer.
- MIDI-VAE presents the first successful application of unaligned style transfer to musical compositions, changing existing songs between styles such as Classical and Jazz.
- The model uses parallel VAEs with a shared latent space and an additional style classifier to encode and manipulate musical style.
- MIDI-VAE handles harmonic polyphonic music with multiple instruments while modeling note durations and velocities as musical dynamics.
2. RELATED WORK
Prior work established neural style transfer and generative modeling for images, text, symbolic music, and raw audio. MIDI-VAE builds on this landscape while targeting style transfer across complete musical compositions.
- Neural style transfer began with image models and expanded to sequential data through autoencoder methods for changing sentence sentiment or content.
- Existing work includes changing musical styles or instrument sounds, but MIDI-VAE extends the target to dynamics, instrumentation, pitches, medleys, interpolations, and song mixtures.
- Earlier music systems used Markov models and recurrent neural networks to compose music in particular styles.
- Variational autoencoders and GANs have become effective for symbolic music generation, including hierarchical modeling and interpolation of polyphonic music.
- Raw-audio generators such as WaveNet and SampleRNN are powerful but face the high dimensionality of audio, so most music-generation work uses symbolic representations.
3. MODEL ARCHITECTURE
MIDI-VAE represents multitrack MIDI through pitch, velocity, and instrument information, then models their joint distribution with parallel recurrent VAEs sharing a latent space. A style classifier organizes selected latent dimensions so encoded songs can be decoded in another style.
- 3.1 Symbolic Music Representation: MIDI files are divided into bars and represented with pitch, velocity, and instrument tracks, extending piano rolls to capture dynamics and instrumentation.
- 3.1 Symbolic Music Representation: Unrolling piano rolls through time lets the recurrent network learn the joint distribution of simultaneously played notes, reducing independently sampled dissonant combinations.
- 3.2 Parallel VAE with Shared Latent Space: The VAE loss combines reconstruction with a KL-divergence term that keeps latent variables close to a chosen prior, enabling generation by sampling latent vectors.
- 3.2 Parallel VAE with Shared Latent Space: The architecture uses separate recurrent encoder-decoder pairs for pitch, velocity, and instrument rolls, all sharing a latent space that captures their joint distribution.
- 3.3 Style Classifier: Increasing β can improve latent disentanglement but negatively affects reconstruction performance, motivating the additional style-classifier structure.
- 3.3 Style Classifier: A style classifier attached to the top k latent dimensions writes a compact style label, and swapping those dimensions changes a song from one genre or composer to another.
4. IMPLEMENTATION
The implementation uses MIDI bars and recurrent architectures to train genre-pair models, with hyperparameter searches focused on pitch-roll reconstruction. Final models use β = 0.1 and σϵ = 0.01 and are trained until decoder convergence.
- Data preprocessing: Songs use multiple instrument tracks, highest-voice selection, 16th-note timing, and a restricted 60-pitch range from C1 to C6.Drum tracks are excluded, and additional voices are selected when songs contain fewer than nT tracks.
- Data preprocessing: The dataset is split into one-bar samples, while recurrent states persist across bars to preserve each song’s sequential order.Songs are shuffled between epochs, but their bars remain ordered and RNN states are not reset between samples.
- Model hyperparameters: Reducing σϵ improves reconstruction and permits higher β because large β performs poorly and σϵ = 1 adds substantial early-learning noise.The authors suggest annealing both hyperparameters might improve results, but they did not test it.
- Model hyperparameters: The final models use β = 0.1 and σϵ = 0.01, with generation sampling z from the empirical latent variance of the training set.The empirical latent mean is reported to be very close to zero.
- Training and evaluation: Four genre-pair models are trained on single GPUs for about 400 epochs, or 48 hours, until the pitch-roll decoder converges.The models are CvJ, CvP, JvP, and BvM; final train/test performance is reported in Table 2.
5. EXPERIMENTAL RESULTS
MIDI-VAE transfers musical style by changing pitch, velocity, and instrumentation while retaining much of the original melody, and it also supports latent-space analysis and smooth musical interpolation.
- 5.1 Style Transfer: The ensemble classifier evaluates style transfer through before-and-after style likelihoods, with larger source-style decreases indicating stronger transfer.Three classifiers use pitch, velocity, and instrument rolls, and their predictions are combined into a voting ensemble.
- 5.1 Style Transfer: Style transfer changes pitch and velocity toward the target style while retaining most of the original melody and often adapting accompanying instruments.Instrumentation correlates most strongly with style, while pitch and velocity contribute more equally to the change.
- 5.1 Style Transfer: Instrument changes are strongest for Classic-to-Jazz and Classic-to-Pop transfers, whereas pairs with overlapping instrumentation show less pronounced switching.For Jazz-to-Classic conversion, most instruments map to one or multiple alternatives, while piano, ensemble, and reed are rarely changed.
- 5.2 Latent Space Evaluation: Latent vectors separate Jazz and Classic bars in t-SNE space, although some songs overlap and receive uncertain ensemble-style predictions.Bars from the same song tend to cluster closely, likely because the instrument roll remains constant.
- 5.2 Latent Space Evaluation: Latent dimensions are entangled: changing one dimension can affect note counts, pitch statistics, specific pitches, and style-related measures.Higher β values slightly improve disentanglement but strongly reduce reconstruction accuracy.
- 5.3 Generation and Interpolation: Linear interpolation produces musically consistent bridges between bars and can mix entire songs while preserving harmonic character.Interpolated pitch ranges and velocities shift toward the target bar, and original songs can sometimes remain identifiable in mixtures.
6. CONCLUSION
MIDI-VAE performs style transfer between musical compositions while incorporating dynamics and instrumentation, and the authors release code and data to support further research.
- 6. CONCLUSION: MIDI-VAE performs style transfer between musical compositions while modeling velocity, note durations, and instrumentation.The authors describe the model as simple but effective and report evaluations across several datasets with audio examples.
- 6. CONCLUSION: The authors plan a hierarchical extension for longer-timescale style features and larger generated pieces.Code and data are made publicly available to facilitate future research on symbolic-music style transfer and sequence tasks.