Source-linked AI summary
A Hierarchical Latent Vector Model for Learning Long-Term Structure in Music
Adam Roberts, Jesse Engel, Colin Raffel, Curtis Hawthorne, Douglas Eck
TL;DR
Existing recurrent VAEs have difficulty modeling long-term structure in very long sequences, limiting their application to sequential data. The paper introduces MusicVAE, a recurrent VAE with a hierarchical decoder that encourages latent-code use, and reports substantially better performance than a flat baseline on musical sequences. The approach supports improved sampling, interpolation, and reconstruction while retaining a single latent vector for each sequence.
Problem
Recurrent VAEs struggle with very long sequences because powerful autoregressive decoders may ignore the latent code and compressing an entire sequence into one vector becomes difficult.
Method
MusicVAE uses a hierarchical recurrent decoder within a sequential VAE, encoding each sequence into a single latent vector and generating subsequences through decoder-produced embeddings.
Results
MusicVAE achieves substantially better sampling, interpolation, and reconstruction performance than a flat baseline on musical data.
Takeaways & Limitations
A hierarchical decoder can improve recurrent VAE modeling of long-term structure in musical sequences while preserving latent-space creative applications.
Takeaways & Limitations
The paper’s demonstrated application is musical-note sequences, and it identifies posterior collapse as a remaining possibility for autoregressive RNN decoders.
Abstract
from arXiv · showhide
The Variational Autoencoder (VAE) has proven to be an effective model for producing semantically meaningful latent representations for natural data. However, it has thus far seen limited application to sequential data, and, as we demonstrate, existing recurrent VAE models have difficulty modeling sequences with long-term structure. To address this issue, we propose the use of a hierarchical decoder, which first outputs embeddings for subsequences of the input and then uses these embeddings to generate each subsequence independently. This structure encourages the model to utilize its latent code, thereby avoiding the "posterior collapse" problem, which remains an issue for recurrent VAEs. We apply this architecture to modeling sequences of musical notes and find that it exhibits dramatically better sampling, interpolation, and reconstruction performance than a "flat" baseline model. An implementation of our "MusicVAE" is available online at http://g.co/magenta/musicvae-code.
1. Introduction
Deep latent variable models offer useful latent representations and creative operations, but recurrent VAEs struggle with very long discrete sequences. MusicVAE addresses this gap with a hierarchical recurrent decoder and applies it to the long-term structure of musical sequences.
- 1. Introduction: VAEs explicitly model latent variables and data likelihoods, supporting inference, sampling, and mappings between latent and data spaces.The latent vector is intended to capture pertinent characteristics and disentangle factors of variation.
- 1. Introduction: Latent-space operations such as averaging and attribute-vector manipulation can produce targeted changes to generated data.These properties motivate applying deep latent variable models to creative machine-learning applications.
- 1. Introduction: Recurrent latent variable models have had limited success on sequential data, especially very long sequences of discrete tokens, because autoregressive decoders may ignore latent codes.The limitation is particularly relevant to musical scores, which typically require autoregressive decoding.
- 1. Introduction: MusicVAE introduces a hierarchical recurrent decoder to address long-term structure while encoding an entire sequence into one latent vector.The model is experimentally reported to autoencode substantially longer sequences than a flat decoder RNN baseline.
- 1. Introduction: Music provides a demanding testbed because popular music contains hierarchical repetition and variation across sections, measures, and beats.It is also a multistream signal with dependencies among players.
- 1. Introduction: The paper evaluates MusicVAE through quantitative and qualitative experiments on musical data.The stated evaluation focuses on demonstrating the method’s ability to model musical sequences.
2. Background
VAEs combine reconstruction with prior-regularized latent representations, but recurrent versions face posterior collapse and struggle to compress long sequences. MusicVAE addresses these issues with a hierarchical recurrent decoder and evaluates latent-space properties for creative applications.
- Variational Autoencoders: VAEs train an encoder and decoder to reconstruct inputs while keeping the approximate posterior close to a prior.The encoder approximates q(z|x), while the decoder parameterizes p(x|z).
- Variational Autoencoders: The ELBO balances reconstruction quality against latent-code regularization, creating a trade-off between information rate and data distortion.The reconstruction term promotes accurate outputs, while the KL term encourages realistic samples from the prior.
- Variational Autoencoders: β weighting and free bits modify KL regularization to adjust the reconstruction–representation trade-off, though these objectives no longer optimize a likelihood lower bound.β < 1 prioritizes reconstruction, while thresholding provides a budget of free bits before KL regularization applies.
- Latent Space Manipulation: Latent-space interpolation tests whether points between two encoded examples decode to realistic, semantically meaningful, and locally smooth outputs.The paper uses spherical interpolation in practice because the VAE prior is a high-dimensional spherical Gaussian.
- Latent Space Manipulation: Attribute vectors average latent codes across examples sharing an attribute, enabling attribute-directed changes in decoded data.The supplied passage introduces attribute-vector construction and its use for testing whether the model discovers attributes.
- Recurrent VAEs: A recurrent VAE encodes an entire sequence into one latent vector and initializes an autoregressive decoder RNN from that vector.Training combines input reconstruction with matching qλ(z|x) to the prior p(z).
- Recurrent VAEs: Powerful autoregressive decoders may ignore the latent code, while compressing increasingly long sequences into one vector becomes difficult.These issues motivate the hierarchical decoder introduced by MusicVAE.
3. Model
MusicVAE extends a sequential VAE with a hierarchical recurrent decoder that supplies embeddings to independently generated subsequences. Restricting bottom-level recurrence to each subsequence makes the conductor embeddings the decoder’s route to longer-term context, while multi-stream decoding supports separate instruments.
- Hierarchical decoder: The model follows sequential VAE structure but replaces the flat decoder with a hierarchical RNN designed for long sequences.The authors describe the hierarchical decoder as the model’s novel architectural enhancement and report substantially better long-sequence performance.
- Bidirectional encoder: A bidirectional LSTM encoder processes the entire input sequence and produces latent-distribution parameters from its final forward and backward states.The encoder uses fully connected layers to produce µ and σ; experiments use 2048-unit LSTM layers and 512 latent dimensions.
- Hierarchical decoder: The decoder first transforms z into a conductor state, whose recurrent outputs provide one embedding for each segmented subsequence.The conductor is initialized from z through a fully connected layer and tanh activation, then produces U embeddings.
- Hierarchical decoder: Each conductor embedding initializes a shared bottom-level decoder, which autoregressively generates token distributions for its corresponding subsequence.The current conductor embedding is concatenated with the previous output at each bottom-level decoding step.
- Hierarchical decoder: Limiting bottom-level state propagation to individual subsequences forces longer-term information to pass through conductor embeddings rather than persist through one unlimited recurrent stream.This design addresses the vanishing influence of the latent state observed with a simple RNN decoder, although posterior collapse remains possible in principle.
- Multi-stream modeling: The trio variant treats instruments as another hierarchy dimension, using separate decoder RNNs for drum, bass, and melody streams.The conductor embeddings initialize the instrument-specific recurrent decoders through separate transformations.
4. Related Work
MusicVAE differs from related sequential latent-variable models primarily through its hierarchical decoder and its use of one latent variable for an entire sequence. This distinction preserves sequence-level operations such as sampling and interpolation while targeting the degraded long-sequence behavior of flat recurrent decoders.
- Recurrent VAEs: Relative to recurrent VAEs, MusicVAE’s primary distinction is its hierarchical RNN decoder rather than a flat RNN decoder.The flat RNN baseline exhibits significantly degraded performance on very long sequences.
- Autoregressive VAE variants: Other VAE extensions combine latent-variable models with feed-forward, convolutional, PixelCNN, or PixelRNN autoregressive decoders, mainly in sequence or image settings.These approaches include recurrent-VAE extensions, PixelVAE, and the Variational Lossy Autoencoder.
- Hierarchical autoencoders: Hierarchical paragraph autoencoders also use recurrent hierarchy, but lack constraints on the latent code and therefore cannot sample or interpolate between sequences.MusicVAE otherwise differs in its encoder design and latent-variable formulation.
- Sequence-to-sequence models: Sequence-to-sequence models likewise compress an input into a representation used to condition generation, including NSynth’s compressed audio embeddings and WaveNet-style reconstruction.MusicVAE can be viewed broadly within this sequence-to-sequence framework.
- Stochastic sequential models: Models with stochastic recurrent states and hierarchical latent variables differ from MusicVAE because they assign latent variables across time or subsequences rather than one variable to the entire sequence.The single sequence-level latent variable supports interpolation and attribute manipulation.
5. Experiments
Experiments evaluate MusicVAE on short and long musical sequences using reconstruction, interpolation, attribute manipulation, and listening tests. The hierarchical decoder substantially improves long-sequence reconstruction, latent-space coherence, attribute control, and perceived sample quality over the flat baseline.
- Data and training: The study uses MIDI-derived 2- and 16-bar melodies, drum patterns, and multi-stream trio sequences, with reconstruction evaluated on held-out data.The dataset contains approximately 1.5 million unique MIDI files; trio sequences combine melody, bass, and drums.
- Short sequences: Short-sequence flat recurrent VAEs reconstruct accurately, use their latent codes, and show only a few-percent gap between teacher-forced and sampled accuracy.These models also produced compelling samples and interpolations without posterior collapse or exposure bias.
- Long-sequence reconstruction: More than 27% discrepancy between teacher-forced and sampled reconstruction accuracy occurred for flat 16-bar models, motivating the hierarchical decoder.The flat model could not reliably reconstruct sequences with T = 256.
- Interpolations: Hierarchical latent interpolations morph smoothly between sequences and remain as probable as endpoint melodies across the interpolation range, unlike naive data blending.Midpoint examples combine semantic properties of both melodies while avoiding the harmonic and rhythmic dissonance of data-space mixing.
- Listening tests: Listening tests preferred hierarchical-model samples dramatically more often than flat-model samples and found statistically significant differences for melodies, trios, and drums.The reported Kruskal-Wallis tests were significant for melodies, trios, and drums, each with p < 0.001.
6. Conclusion
The paper concludes that MusicVAE uses a hierarchical decoder to model long-term structure in musical sequences, achieving substantially better performance than a flat baseline. The authors also identify testing on other sequential data and releasing code and pretrained models as future-facing steps.
- MusicVAE combines a recurrent variational autoencoder with a hierarchical decoder to model long-term structure.
- Experiments on music data showed substantially better quantitative and qualitative performance than a flat baseline.
- The authors propose testing MusicVAE on other sequential data in future work.
- The code and pretrained models were made publicly available to facilitate research on recurrent latent variable models.
A. Dataset Creation Details
The dataset was created from publicly available MIDI files, filtered and quantized into bar-based musical sequences. Sliding windows produced melody, drum, and trio examples, with held-out data reserved for evaluation.
- ≈1.5 million unique MIDI files were collected from the web as the source corpus.
- Files with non-4/4 time signatures were removed, and tempo-defined bar boundaries were quantized to 16 notes per bar.
- Sliding windows extracted 2- and 16-bar drum sequences, producing 3.8 million and 11.4 million examples, respectively.
- Sliding windows extracted 2- and 16-bar monophonic melodies, producing 28.0 million and 19.5 million unique examples, respectively.
- Trio sequences combined melody-range, bass-range, and drum instruments, considering all category combinations when multiple instruments qualified.
- Held-out examples were reserved for reporting reconstruction accuracy and interpolation results.
B. Lakh MIDI Dataset Results
The Lakh MIDI Dataset served as a separate training and evaluation source for 16-bar models. Results closely matched those from the full dataset, preserving the same relative conclusions about model performance.
- The Lakh MIDI Dataset supplied 3.7 million melodies, 4.6 million drum patterns, and 116 thousand trios for 16-bar modeling.
- The LMD-trained melody model differed by less than 1% in reconstruction accuracy from the original model on an evaluation subset excluding LMD training examples.
- LMD-trained models achieved slightly higher accuracies than Table 1 while preserving the same relative performance conclusions.
- Table 2 reports held-out reconstruction accuracies under both teacher-forcing and full sampling.
C. Attribute Definitions
This section states that definitions were used to measure the amount of each attribute, but the supplied passage does not specify those definitions.
- Definitions were used to measure the amount of each attribute.
- The passage identifies attribute measurement as the purpose of the definitions.
- No individual attribute definitions are specified in the supplied passage.
C Diatonic
This section defines five note-sequence attributes used to characterize musical structure, covering scale membership, density, pitch movement, and two forms of syncopation.
- C Diatonic measures the fraction of notes whose pitches lie in the C diatonic scale, A-B-C-D-E-F-G.
- Note Density measures note onsets divided by sequence length in 16th-note steps.
- Average Interval is the mean absolute pitch interval between consecutive notes.
- 16th Note Syncopation measures odd 16th-note onsets lacking an onset at the immediately previous 16th-note position.
- 8th Note Syncopation measures odd 8th-note onsets lacking onsets at the previous 16th- or 8th-note positions.
D. Audio Samples
Synthesized audio accompanies all examples in this section and the main text, with recordings provided in the online supplement.
- Synthesized audio is available for all examples presented in this section.
- The audio supplement also covers examples appearing in the main text.
- The online supplement is the location for accessing the synthesized audio.
E. Additional Figures and Samples
The additional figures illustrate attribute-vector manipulations, latent and data-space interpolations, resampling, and generated 16-bar music samples. They also provide a 2-bar melody interpolation with pitch plotted over time.
- Attribute vectors: Figure 6 varies the Note Density attribute-vector amount from -1.5 to 1.5 in 0.5 steps, with no vector at the center.
- Attribute vectors: Figure 7 resamples the same latent code, producing semantically similar sequences whose specific notes vary because of autoregressive-decoder sampling.
- Attribute vectors: Figures 8-11 add or subtract attribute vectors for C Diatonic, Average Interval, 16th Note Syncopation, and 8th Note Syncopation.
- Interpolation: Figures 12 and 13 interpolate between the same endpoint sequences in data space and MusicVAE latent space, respectively.
- Interpolation: Figure 14 shows six interpolated 2-bar melodies between test-set endpoints, with pitch from A3 to C8 on the vertical axis and time horizontally.
- Generated samples: Figure 15 presents a selected 16-bar trio sample generated by MusicVAE.