Source-linked AI summary
The challenge of realistic music generation: modelling raw audio at scale
Sander Dieleman, Aäron van den Oord, Karen Simonyan
TL;DR
Realistic music generation must preserve performance nuances while modelling dependencies from waveform-scale detail to musical form. This paper uses autoregressive discrete autoencoders to hierarchically extend autoregressive receptive fields, enabling stylistically consistent raw-audio piano generation across tens of seconds, with fidelity traded against long-range structure.
Problem
Music generation models commonly use symbolic representations that omit performance nuances, while raw-audio autoregressive models struggle to capture musical correlations beyond local structure.
Method
The paper uses autoregressive discrete autoencoders with compressed conditioning signals and hierarchical stages to extend autoregressive receptive fields efficiently.
Results
Hierarchical autoregressive modelling generates stylistically consistent piano music directly as raw audio across tens of seconds, spanning about 400,000 timesteps in one reported setting.
Takeaways & Limitations
Long-range musical structure can be modelled in raw audio with stacked autoregressive stages, but improved large-scale structure comes at some cost to signal fidelity.
Takeaways & Limitations
Scaling receptive fields to musical structure at the scale of minutes would require substantially more model capacity and training data.
Abstract
from arXiv · showhide
Realistic music generation is a challenging task. When building generative models of music that are learnt from data, typically high-level representations such as scores or MIDI are used that abstract away the idiosyncrasies of a particular performance. But these nuances are very important for our perception of musicality and realism, so in this work we embark on modelling music in the raw audio domain. It has been shown that autoregressive models excel at generating raw audio waveforms of speech, but when applied to music, we find them biased towards capturing local signal structure at the expense of modelling long-range correlations. This is problematic because music exhibits structure at many different timescales. In this work, we explore autoregressive discrete autoencoders (ADAs) as a means to enable autoregressive models to capture long-range correlations in waveforms. We find that they allow us to unconditionally generate piano music directly in the raw audio domain, which shows stylistic consistency across tens of seconds.
1 Introduction
Raw-audio music generation preserves performance nuances that symbolic representations abstract away, but requires modelling correlations spanning milliseconds to minutes. The paper addresses this challenge with autoregressive discrete autoencoders and demonstrates stylistically consistent piano generation across long timescales.
- Challenge: Music audio contains structure from millisecond-scale waveform periodicity to musical form spanning several minutes.These correlations span many orders of magnitude, making them difficult to model jointly.
- Challenge: Symbolic representations such as scores and MIDI abstract away performance-specific timing, timbre, and volume variations that affect musicality.Such nuances can be difficult and impractical to capture symbolically for many instruments.
- Challenge: Raw-audio modelling retains musically relevant information and applies across arbitrary instrument combinations and non-musical audio, but is substantially more difficult than symbolic modelling.Waveforms are still somewhat lossy digital representations, while direct audio generation operates at very high sampling rates.
- Prior work: Autoregressive waveform models generate realistic local audio structure, yet previous music applications failed to produce interesting structure at timescales of seconds and beyond.Unlike text-to-speech, music generation lacks strong conditioning that relieves models from representing longer-range structure.
- Result: Hierarchical modelling captures musical audio structure across roughly 400,000 timesteps, or about 25 seconds at 16 kHz, producing stylistically consistent piano samples.The stages are trained separately to mitigate hardware limitations.
- Approach: The paper uses autoregressive discrete autoencoders to enlarge receptive fields efficiently by splitting raw-audio modelling across abstraction levels.It introduces AMAE as an alternative to VQ-VAE and establishes raw-audio music generation as a benchmark for long-range structure.
2 Scaling up autoregressive models for music
Scaling autoregressive models to music requires addressing hardware limits and their bias toward local signal structure. The paper stacks autoregressive models through compressed conditioning signals, enlarging audio-domain receptive fields while trading some information fidelity for long-range musical structure.
- Scaling challenge: Increasing WaveNet dilation or SampleRNN tiers enlarges receptive fields, but training requires excerpts at least as long as those fields and quickly reaches hardware limits.Each second of audio contains many thousands of timesteps, making long-context training expensive.
- Scaling challenge: Autoregressive models are biased toward local structure because recent waveform history is more informative than distant history for predicting a timestep.Capturing long-term correlations therefore requires redistributing capacity, which can reduce signal fidelity while improving musicality.
- Stacking autoregressive models: An encoder learns a high-level conditioning signal, temporal downsampling makes it coarse-grained, and an autoregressive decoder models residual local structure.The conditioning-to-input sample-rate ratio is called the hop size h.
- Stacking autoregressive models: Modelling the downsampled conditioning sequence with another autoregressive model yields an audio-domain receptive field h times larger than its representation-level receptive field.Larger hop sizes increase compression and information loss, so h cannot be chosen arbitrarily.
- Stacking autoregressive models: A variational-autoencoder formulation risks posterior collapse because powerful autoregressive decoders may ignore latent variables under regularisation.This motivates discrete autoencoder designs rather than relying directly on the probabilistic latent-variable formulation.
3 Autoregressive discrete autoencoders
Autoregressive discrete autoencoders use a discrete bottleneck to control representation capacity and provide high-level conditioning for autoregressive waveform decoders. The AMAE replaces learned vector quantisation with simplex-constrained queries and argmax quantisation, improving convergence reliability when VQ-VAE codebooks collapse.
- Discrete representations directly control the information content of an autoencoder bottleneck and make training an autoregressive model on top easier.Continuous representations can pass information through the encoder instead of the autoregressive connections.
- An ADA encodes waveform x into queries q, quantises them into q′, and reconstructs x with a decoder conditioned on q′.This creates a discrete bottleneck between the encoder and decoder.
- 3.1 VQ-VAE: VQ-VAE learns a codebook jointly with the model and combines reconstruction, codebook, and commitment losses.The codebook loss can instead be optimized with exponentially smoothed K-means, which the authors adopt to speed convergence.
- 3.1 VQ-VAE: VQ-VAE codebook collapse can leave part of the discrete bottleneck unused, worsening likelihoods and reconstructions on challenging datasets.Population based training adapts α and β online to mitigate this issue to some extent.
- 3.2 AMAE: AMAE avoids learning a codebook by mapping encoder queries onto the (k −1)-simplex and choosing the nearest k-dimensional one-hot vector with argmax quantisation.A diversity loss encourages all outputs to be used, while ReLU followed by divisive normalisation was found to converge more robustly than softmax.
- 3.2 AMAE: AMAE usually slightly underperforms an equally structured VQ-VAE but converges much more reliably when VQ-VAE suffers from codebook collapse.Adding the VQ-VAE commitment loss produced no noticeable practical difference.
4 Experiments
Experiments compare audio and code-sequence ADAs, showing trade-offs among input representation, compression, codebook stability, receptive field, and perceptual quality. Hierarchical models improve musical structure, although fidelity, convergence, and evaluation reliability remain limiting factors.
- Experimental setup: 413 hours of solo piano recordings provide the evaluation dataset, chosen to reduce timbral variety while retaining polyphonic music.The experiments focus on a single instrument with many high-quality recordings.
- ADA architectures for audio: One-hot input improves multinomial NLL by encoding precise signal values, but continuous-input VQ-VAE reconstructions sound better because one-hot training suffers partial codebook collapse.This separates likelihood quality from perceptual reconstruction quality.
- ADA architectures for audio: AMAE avoids codebook collapse but slightly underperforms VQ-VAE, with worse reconstruction quality and sometimes inconsistent reconstruction volume.The comparison favors AMAE for stability rather than perceptual quality.
- ADA architectures for audio: An 8× larger compression factor with hop size 64 still yields surprisingly good reconstruction quality, motivating architecture #1.2 as the basis for further experiments.The selected setup balances compression with perceptual reconstruction quality.
- ADA architectures for code sequences: Code sequences are harder to predict locally than waveforms, with a clear predictability transition at receptive field 64 matching the encoder’s receptive field.The transition indicates that the encoder’s local context determines how signal information is represented and compressed.
- Multi-level models: Multi-level models improve musical structure: two-level VQ-VAE models raise musicality with little fidelity loss, while three-level models become more musically interesting but lose signal fidelity.Single-level models fail to produce compelling samples, and extending receptive fields alone does not guarantee musicality.
5 Discussion
The approach extends autoregressive receptive fields efficiently to model raw-audio music over tens of seconds, trading some signal fidelity for long-range structure. Further scaling to minute-level musical structure would require substantially more training data.
- Autoregressive models at multiple abstraction levels capture long-range audio correlations across tens of seconds, or 100,000s of timesteps.This comes at the cost of some signal fidelity, indicating a trade-off between local accuracy and large-scale structure.
- The approach addresses raw-audio music generation by extending autoregressive receptive fields in a computationally efficient manner.
- Increasing sample rate and bit depth should improve generated-sample fidelity relatively straightforwardly.
- Learning musical structure at the scale of minutes would require more model capacity and substantially more training data.
A Details of model architecture and training
The model architecture combines WaveNet-based local processing with separately configured encoder, modulator, and large-receptive-field WaveNet components. Training uses Adam and substantial GPU resources, with population-based tuning for VQ-VAE parameters.
- Model architecture: The local ADA model uses a 32-block WaveNet with four repetitions of eight dilation stages, yielding a 1024-timestep or 64 ms receptive field.
- Model architecture: The encoder produces 8-bit codes with 256 symbols and downsamples the sequence by a factor of 8, while the modulator has a 256 ms receptive field.
- Model architecture: The large WaveNet has a 6144-timestep receptive field, whereas the very large version uses 12288 timesteps and takes about four times longer to train.
- Training: The models use Adam with a learning rate of 2 · 10^-4 for 500,000 iterations, and ADAs are trained on 8 GPUs with 16GB RAM each.
- Training: VQ-VAE population-based training uses 20 individuals and perturbs parameters sampled from specified ranges every 5000 iterations after an initial 10000-iteration period.
B Dataset
The dataset contains just under 413 hours of clean solo-piano recordings, with 99% used for training and 1% held out for evaluation. Composer imbalance introduces a bias toward romantic styles.
- Dataset composition: The dataset contains just under 413 hours of clean solo-piano recordings in 16-bit PCM mono format sampled at 16 kHz.
- Dataset composition: The same composition may appear multiple times as different performances, while live recordings were filtered for applause and excessive background noise.
- Dataset composition: The dataset lists the composers whose work it contains, with composer identities represented in Table 4.
- Dataset limitations: Popular composers such as Chopin, Liszt, and Beethoven are overrepresented, and many samples audibly show a bias toward romantic composers.