Source-linked AI summary
Symbolic Music Generation with Diffusion Models
Gautam Mittal, Jesse Engel, Curtis Hawthorne, Ian Simon
TL;DR
Diffusion models have been limited on discrete sequential data, including symbolic music, because their iterative refinement operates in continuous domains. The paper models continuous latents from a pretrained VAE with a diffusion model, achieving strong unconditional generation and post-hoc conditional infilling for symbolic music. Its main scope is demonstrated by the model’s dependence on VAE latent quality and the stated infilling baseline limitation.
Problem
Diffusion models had not been applied to symbolic music because iterative refinement was confined to continuous domains, limiting their use for discrete sequential data.
Method
The method trains a diffusion model over continuous latents produced by a low-level VAE to model long discrete MIDI sequences.
Results
The model produces high-quality unconditional 1024-token melodic sequences, outperforms TransformerMDN in hierarchical latent modeling, and supports post-hoc conditional infilling.
Takeaways & Limitations
Diffusion models can serve as higher-level priors for discrete symbolic music and support both unconditional generation and conditional infilling.
Takeaways & Limitations
Sampling becomes more difficult for VAEs with better reconstructions because latent holes arise in the approximate posterior, and the autoregressive baseline cannot condition on final embeddings during infilling.
Abstract
from arXiv · showhide
Score-based generative models and diffusion probabilistic models have been successful at generating high-quality samples in continuous domains such as images and audio. However, due to their Langevin-inspired sampling mechanisms, their application to discrete and sequential data has been limited. In this work, we present a technique for training diffusion models on sequential data by parameterizing the discrete domain in the continuous latent space of a pre-trained variational autoencoder. Our method is non-autoregressive and learns to generate sequences of latent embeddings through the reverse process and offers parallel generation with a constant number of iterative refinement steps. We apply this technique to modeling symbolic music and show strong unconditional generation and post-hoc conditional infilling results compared to autoregressive language models operating over the same continuous embeddings.
1. INTRODUCTION
The paper adapts diffusion models to discrete symbolic music by modeling continuous VAE latents, enabling long-sequence generation, strong unconditional sampling, and post-hoc conditional infilling.
- Diffusion models had not been applied to symbolic music because their iterative refinement sampling was confined to continuous domains.
- The proposed approach trains DDPMs on continuous latents from a low-level VAE to overcome this limitation.
- The model generates 1024-token discrete melodic sequences through iterative refinement of lower-level VAE latents.
- DDPMs outperform TransformerMDN in hierarchical modeling of continuous latents, partly because they avoid teacher forcing and exposure bias during training.
- The method supports post-hoc conditional infilling of melodic sequences for creative applications.
2. BACKGROUND
Diffusion models use forward noising and reverse denoising processes, while VAEs provide continuous latent representations for discrete sequential data. VAEs trade reconstruction quality against alignment with a Gaussian prior, affecting sampling difficulty.
- Denoising Diffusion Probabilistic Models: DDPMs define same-dimensional latent variables and transform data into Gaussian noise through a fixed forward diffusion process.The forward process adds Gaussian noise over N steps according to a noise schedule.
- Denoising Diffusion Probabilistic Models: The reverse process uses a learned Markov chain to iteratively refine Gaussian noise into data.At each reverse step, the model gradually denoises the current sample.
- Denoising Diffusion Probabilistic Models: The forward process can be evaluated at any step, enabling training on noisy-data pairs with a squared L2 objective.This objective resembles denoising score matching and was found to yield higher-quality samples.
- Variational Autoencoders: VAEs combine an encoder that approximates the posterior with a decoder that models data conditioned on latent codes.The latent code is typically constrained toward an isotropic Gaussian prior.
- Variational Autoencoders: VAE training trades reconstruction quality against closeness between the approximate posterior and Gaussian prior, creating latent holes that make sampling harder.This trade-off is identified as a primary shortcoming of VAEs.
3. MODEL
The model combines a short-sequence MusicVAE with diffusion over its continuous latents to generate long MIDI sequences. Iterative refinement supports unconditional generation and post-hoc infilling without retraining.
- Architecture: The architecture pairs a 2-bar MusicVAE with diffusion over k = 32 latents to model 64-bar MIDI sequences.The diffusion model captures temporal relationships among the VAE latents, extending modeling beyond the VAE's short training sequences.
- Architecture: MusicVAE encodes each 2-bar phrase into continuous latent embeddings that are linearly scaled to [−1, 1] before diffusion training.The phrase vocabulary contains 90 tokens, including note-on, note-off, and 88 pitches.
- Architecture: A transformer diffusion network models sequences of 32 preprocessed 42-dimensional latent embeddings and captures their temporal context with positional encoding.The network uses six encoder layers with eight self-attention heads each.
- Architecture: Continuous noise conditioning and feature-wise linear modulation condition residual layers on the diffusion noise level.The model uses a linear schedule with N = 1000 steps, β1 = 10^-6, and βN = 0.01.
- Unconditional Generation: The model targets long-term structure in unconditional generation by learning dependencies among latent embeddings, even when the VAE posterior differs substantially from its Gaussian prior.This latent-space modeling is described as improving sample quality.
- Infilling: Iterative reverse sampling can be steered and conditioned after training, enabling latent infilling for partially occluded musical sequences.The procedure diffuses fixed regions of a partial sample and uses a mask while the reverse process generates occluded regions.
4. METHODS
Experiments use LMD-derived monophonic sequences encoded by MusicVAE, comparing the diffusion model with a matched autoregressive TransformerMDN baseline. Evaluation combines latent-distribution distances with framewise pitch and duration self-similarity metrics.
- Data: 988,893 training and 11,295 validation 64-bar monophonic sequences were extracted from the Lakh MIDI Dataset and encoded into 32 MusicVAE latent embeddings.The dataset contains over 170,000 MIDI files, with 99% used for training.
- Baseline: TransformerMDN uses the same architecture as the diffusion model before its output layer, but has 38M parameters versus 25.58M because of its 100-Gaussian mixture-density output.Both models are trained on the same dataset for a fair comparison.
- Optimization: The diffusion model was trained for 500K steps with batch size 64, while TransformerMDN used teacher forcing for 250K steps with batch size 128.Both models were trained for 6.5 hours on a single NVIDIA Tesla V100 GPU, using different learning rates.
- Self-similarity metric: Framewise Overlapping Area measures local self-similarity by comparing adjacent four-measure windows with two-measure hops across generated sequences.Pitch and duration statistics are modeled separately within each frame.
- Self-similarity metric: Pitch and duration consistency and variance are aggregated across adjacent frames and normalized relative to the training set, with errors exceeding 100% clipped to zero similarity.The metric uses Gaussian statistics and the Overlapping Area of adjacent frames.
- Latent-space evaluation: Fréchet distance and polynomial-kernel Maximum Mean Discrepancy measure differences between generated and original continuous latent distributions.These metrics do not measure long-term temporal consistency or output quality.
5. RESULTS
The diffusion model produces high-quality unconditional symbolic-music samples and strong conditional infilling, outperforming the evaluated baselines on framewise similarity metrics. Iterative refinement improves sample quality, while latent-space metrics provide an incomplete view of generation quality.
- Unconditional Generation: The diffusion model produces samples most similar to training data on relative framewise overlapping-area metrics for note pitch and duration.
- Unconditional Generation: The diffusion model outperforms TransformerMDN, whose autoregressive latent modeling is challenged by high dimensionality and teacher-forcing exposure bias.
- Unconditional Generation: The Gaussian prior has low consistency and high variance, whereas interpolation has low variance and excessive consistency from repetition.
- Unconditional Generation: Latent-space evaluations favor TransformerMDN, while their distributional assumptions limit coverage of the full latent manifold.
- Unconditional Generation: Latent metrics improve steadily during refinement, while consistency similarity starts high and variance similarity emerges only near the end.
- Infilling: The diffusion model outperforms interpolation and independent-prior baselines in both consistency and variance similarity for conditional infilling.
6. RELATED WORK
Related work covers multi-stage generative systems, iterative refinement in discrete musical spaces, and latent-space conditioning of unconditionally trained models. The paper distinguishes its approach by using a VAE to parameterize musical notes for diffusion modeling.
- Multi-stage learning: Multi-stage systems model intermediate representations before generating final symbolic, audio, or image outputs.
- Iterative refinement: Prior iterative-refinement methods generate musical content directly in discrete spaces using Gibbs sampling, gradient-based sampling, or score-based models.
- Iterative refinement: The approach differs by using a VAE to parameterize the discrete musical-note space for DDPM generation.
- Conditional sampling: Related work steers generation in the latent space of an unconditionally trained model, often by adding a model over a pretrained VAE.
7. CONCLUSION
The paper demonstrates a multi-stage model that combines a low-level VAE with a higher-level diffusion model over continuous latents. It supports symbolic-music generation in both unconditional and conditional-infilling settings.
- A low-level VAE supplies continuous latents that a higher-level diffusion model uses to generate long-form discrete symbolic music.
- The demonstrated approach is useful for both unconditional generation and conditional infilling of symbolic music.
- Future work will extend the approach to other discrete data and explore more post-hoc conditioning methods for creative applications.
A. DDPM TRAINING AND SAMPLING
The DDPM training and sampling procedure uses a fixed noise schedule to perturb data and a reverse process to denoise samples iteratively. Training samples noisy states and learns denoising updates, while sampling starts from Gaussian noise.
- Training and sampling: Training samples a diffusion step and Gaussian noise, then takes a gradient step using the learned denoising objective.
- Training and sampling: The forward process converts data x_0 into noisy latents through N steps governed by a noise schedule β_1, ..., β_N.
- Training and sampling: Sampling initializes x_N from N(0, I) and iterates reverse steps from t = N to 1.
B. MUSICVAE ARCHITECTURE
MusicVAE uses a bidirectional LSTM encoder and autoregressive LSTM decoder, while latent analysis identifies underused dimensions that are removed before diffusion training. The evaluations compare framewise self-similarity and latent-space behavior across models.
- B. MUSICVAE ARCHITECTURE: The 2-bar MusicVAE uses a bidirectional LSTM encoder and an autoregressive LSTM decoder.
- C. TRIMMING LATENTS: Posterior standard deviations identify latent dimensions that are underutilized because their average variance remains close to the unit Gaussian prior.The red-highlighted region in Figure 5 marks these unused dimensions.
- D. TABLES: The reported evaluations cover unnormalized framewise self-similarity and latent-space metrics for generated samples.Table 2 measures overlapping area, while Table 3 reports latent-space evaluations.
E. ADDITIONAL SAMPLES
The supplementary figures provide piano-roll examples from test data, unconditional generation, interpolation, prior sampling, and conditional infilling. The infilling visualizations hold boundary tokens fixed while generating the interior sequence.
- E. ADDITIONAL SAMPLES: Extended visual and audio samples from each model are available in the online supplement.
- E. ADDITIONAL SAMPLES: Figure 6 shows additional piano rolls from sequences drawn from the test set.
- E. ADDITIONAL SAMPLES: Figures 7–10 show additional unconditional piano rolls from the diffusion model, TransformerMDN, spherical interpolation, and independent MusicVAE-prior sampling.
- E. ADDITIONAL SAMPLES: Figure 11 visualizes infilling experiments that hold the first and last 256 melody tokens constant and fill the interior 512 tokens.The figure compares original samples, diffusion outputs, interpolation, and independent MusicVAE-prior sampling.