Source-linked AI summary
Jukebox: A Generative Model for Music
Prafulla Dhariwal, Heewoo Jun, Christine Payne, Jong Wook Kim, Alec Radford, Ilya Sutskever
TL;DR
Jukebox tackles the computational challenge of generating long, high-quality raw-audio music with singing. It compresses audio into multi-scale discrete codes modeled by autoregressive Transformers, producing diverse, coherent songs with artist, genre, and lyric conditioning. The paper identifies remaining quality limitations across time scales and audio fidelity.
Problem
Long raw-audio music requires modeling millions of samples while capturing musical structure, and prior systems did not provide one system combining diverse music generation with singing.
Method
Jukebox uses separate multi-scale VQ-VAEs to compress raw audio into discrete codes, autoregressive Transformers and upsamplers to model them, and conditioning networks for artist, genre, and lyrics.
Results
Jukebox generates diverse, high-fidelity raw-audio songs with coherence spanning multiple minutes, recognizable singing, and optional artist, genre, and lyric conditioning.
Takeaways & Limitations
The model extends raw-audio music generation to multi-minute pieces with natural-sounding singing and controllable musical and vocal styles.
Takeaways & Limitations
The model is stronger at mid-range time scales, while future work remains to achieve high quality across all time scales and eliminate unwanted noise.
Abstract
from arXiv · showhide
We introduce Jukebox, a model that generates music with singing in the raw audio domain. We tackle the long context of raw audio using a multi-scale VQ-VAE to compress it to discrete codes, and modeling those using autoregressive Transformers. We show that the combined model at scale can generate high-fidelity and diverse songs with coherence up to multiple minutes. We can condition on artist and genre to steer the musical and vocal style, and on unaligned lyrics to make the singing more controllable. We are releasing thousands of non cherry-picked samples at https://jukebox.openai.com, along with model weights and code at https://github.com/openai/jukebox
1. Introduction
Jukebox addresses the gap between specialized music-generation systems by generating diverse, high-fidelity raw-audio music with coherence over multiple minutes. Its hierarchical compression and autoregressive modeling also support stylistic and lyrical control.
- Existing systems separately capture melody, composition, timbre, or singing voice, but a single system combining these capabilities remains elusive.
- Symbolic music generation simplifies modeling but constrains outputs to note sequences and fixed instrument sets.
- Jukebox uses hierarchical VQ-VAE compression, a Sparse Transformer, and autoregressive upsamplers to model raw audio across compression levels.
- The resulting system generates diverse, high-fidelity raw-audio music with long-range coherence spanning multiple minutes.
- The models produce varied genres and capture melody, rhythm, composition, instrumental timbres, and singers’ styles and voices.
- Conditional priors allow generation to be guided by lyrics or MIDI, while priming supports novel completions of existing songs.
2. Background
Raw-audio music generation is computationally difficult because long recordings contain millions of high-resolution samples while requiring modeling from timbre to global coherence. Jukebox makes this tractable by discretizing audio with a one-dimensional VQ-VAE.
- A four-minute audio segment contains approximately 10 million samples, making long-duration raw-audio generation computationally demanding.
- The task must represent musical structures ranging from timbre to global coherence while preserving substantial diversity.
- A one-dimensional VQ-VAE compresses raw audio into a shorter sequence of discrete codes through encoding, vector quantization, and decoding.
- The hop length is the ratio T/S between the input sequence length and the number of latent codes.
- The VQ-VAE objective combines reconstruction, codebook, and commitment losses, with commitment regularization stabilizing encoder representations.
- Hierarchical VQ-VAE representations use multiple codebooks with decreasing sequence lengths to encode information at different abstraction levels.
3. Music VQ-VAE
Jukebox uses separate raw-audio VQ-VAEs at multiple temporal resolutions to maintain information across abstraction levels. Random restarts, spectral loss, and multi-resolution spectral objectives address codebook utilization and frequency reconstruction.
- Three separate raw-audio VQ-VAEs encode audio at different temporal resolutions, using WaveNet-style dilated convolutions with downsampling and upsampling.
- Random restarts reset underused codebook vectors to encoder outputs, ensuring vectors remain utilized and can continue learning.
- The hierarchy represents longer audio per token at higher levels, while bottom-level codes provide the highest-quality reconstruction.
- Separate autoencoders avoid the top-level collapse observed when lower levels carry most of the information in a hierarchical raw-audio VQ-VAE.
- 3.3. Spectral Loss: Sample-level reconstruction alone favors low frequencies, so spectral loss is added to capture mid-to-high-frequency information.
- 3.3. Spectral Loss: The spectral objective sums losses computed over multiple STFT settings that trade off time and frequency resolution.
4. Music Priors and Upsamplers
Jukebox learns autoregressive priors over hierarchical VQ-VAE codes, using upper-level codes and conditioning signals to generate controllable music. Lyrics conditioning addresses weak word intelligibility by modeling lyric–singing alignment, while windowed and primed sampling extend or continue generation.
- Music Priors and Upsamplers: Separate autoregressive Transformers model the top-level prior and conditional upsamplers for middle- and bottom-level VQ-VAE codes.The lower-level models condition on corresponding upper-level code segments, with sparse-attention Transformers used in the discrete token space.
- Music Priors and Upsamplers: Artist, genre, and timing signals make generation more controllable and allow steering toward selected musical styles.These labels reduce prediction entropy for a particular style and provide control at generation time.
- Lyrics Conditioning: Lyrics-conditioned models address largely unintelligible singing by conditioning on lyrics corresponding to each audio segment.The lyrics-to-singing task must model alignment, vocal identity, and variation caused by pitch, melody, rhythm, and genre.
- Lyrics Conditioning: An encoder-decoder model encodes lyric characters and lets music-token decoder layers attend to lyric features.The decoder produces top-level music tokens, while the lyrics encoder is trained with an autoregressive modeling loss.
- Sampling: Windowed sampling generates segments longer than the context length by repeatedly continuing overlapping windows, trading speed for quality through hop length.Primed sampling instead initializes generation with codes from an actual song to create novel continuations.
5. Experiments
Experiments show that Jukebox generates diverse, coherent music with controllable singing, while reconstruction and musical quality remain limited by compression, model capacity, and missing long-term structure.
- Samples: A larger and more diverse song dataset enabled samples beyond classical music, including varied genres, instruments, melodies, rhythms, timbres, and singing styles.The model also generated novel song completions and could produce new lyrics, although difficult words benefited from phonetic spelling.
- Samples: A 5-billion-parameter top-level prior improved musicality, coherence, and initial singing, while lyric conditioning made singing more intelligible and controllable.The 1-billion-parameter prior was insufficient for singing and diverse musical timbres, and the unconditional model still struggled with recognizable words.
- Samples: Samples remained musically coherent for approximately 24 seconds of top-level context and preserved similar harmonies and textures when generating longer audio.Because the top-level prior lacked the entire song context, generations did not develop recurring choruses or long-term melodic patterns.
- Samples: The model produced varied continuations from the same artist and lyrics, but re-renditions and completions remained lower quality than original songs.Some generations initially resembled originals before deviating into new material after about 30 seconds.
- VQ-VAE Ablations: Higher compression increased reconstruction error and removed high-frequency information, while spectral loss preserved more information but introduced scratchy artifacts.Randomly restarting dead codes improved early codebook usage, and larger codebooks improved reconstruction fidelity.
- VQ-VAE Ablations: A single hierarchical raw-audio VQ-VAE struggled to place information in middle and top levels, whereas separate autoencoders were cleaner and more effective.The experiments suggest that explicitly modeling residuals can push more information to higher levels, but the separate-autoencoder design performed better.
6. Related Work
Jukebox builds on generative modeling for symbolic music, raw audio, and speech, extending hierarchical discrete representations toward long-form raw-audio music generation.
- Earlier music-generation systems commonly operated on symbolic representations such as pianorolls, while later work modeled raw audio or spectrograms.These approaches include rule-based, data-driven, adversarial, autoregressive, and neural synthesis methods.
- VQ-VAE compresses extremely long audio contexts into shorter discrete latent encodings and has been extended with hierarchical representations.The cited work demonstrates applications to images, audio, and unsupervised phoneme representations.
- Jukebox combines hierarchical discrete audio representations with autoregressive modeling to address long-context raw-audio generation.This positions the work at the intersection of VQ-VAE-based compression, raw-audio generation, and Transformer modeling.
- Music-synthesis research also includes systems for generating or transforming singing by modeling vocal timbre, pitch, and expressive control.Earlier concatenative and statistical parametric methods impose strong assumptions and can introduce artifacts, while neural approaches provide alternative synthesis strategies.
7. Future work
The paper identifies future work in improving musical structure, audio fidelity, language and style coverage, human control, and generation speed.
- The model is stronger at mid-range musical structure than at maintaining traditional large-scale forms or consistently clean fine-scale audio.Samples can remain consistent over longer spans, but may lack repeating choruses or question-and-answer melodies and may contain noise or scratchiness.
- The current training data is limited to songs whose primary detected language is English, motivating expansion to other languages and artists.The authors also seek unusual blends of existing musical styles.
- Current lyric and MIDI conditioning provides some steering, but future interfaces could control mood, dynamics, and when particular instruments or voices play.The authors frame broader controllability as important for serving musicians and people without formal training.
- Generation is slow: producing 1 minute of top-level tokens takes around an hour, while upsampling one minute currently takes around 8 hours.The authors propose faster generation and Transformer upsampling, including possible distillation into a parallel sampler.
8. Conclusion
Jukebox generates raw-audio music in diverse artist and genre styles, optionally conditioned on lyrics, using hierarchical compression and autoregressive modeling to produce pieces lasting multiple minutes.
- Jukebox generates raw-audio music imitating many different styles and artists, with optional conditioning on specific artists, genres, and lyrics.The paper describes the hierarchical VQ-VAE compression needed to represent the music as tokens.
- Unlike previous raw-audio music systems producing 20–30 second samples, Jukebox generates pieces that are multiple minutes long.
A. Scalable Transformer
The Scalable Transformer simplifies sparse attention and training machinery so large autoregressive models can operate more efficiently on long token sequences.
- The implementation replaces specialized sparse-attention CUDA kernels with simpler axis-aligned attention patterns using Python-level array manipulation.The patterns include masked row, masked column, and unmasked previous-row attention.
- The model uses a shared initialization scale rather than fan-in and depth rescaling, reducing implementation complexity.
- Fully half-precision training and dynamically scaled optimizer states reduce the memory footprint needed for large models.The optimizer state stores scaling information alongside half-precision tensors.
- Axis-aligned attention reshapes sequences into blocks and block lengths to provide factorized sparse attention for autoregressive modeling.The approach is designed to retain comparable performance while simplifying implementation.
- The attention design supports autoregressive generation while using only simple Python-level array operations.
B.1. Music VQ-VAE
The music VQ-VAE uses three independently trained raw-audio autoencoders at different temporal resolutions, while autoregressive priors and upsamplers model their discrete codes with conditioning on musical metadata and, at the top level, lyrics.
- Music VQ-VAE: Three raw-audio VQ-VAEs produce discrete codes at varying hop sizes for the bottom, middle, and top priors.The autoencoders use non-causal, dilated 1-D convolutions and are trained independently with nonautoregressive reconstruction losses.
- Music VQ-VAE: The middle- and top-level residual networks have receptive fields of 120 ms and 480 ms, while doubling bottom-level residual blocks expands its receptive field to about 2 seconds per code.The actual bottom-level receptive field is mostly local despite its larger nominal span.
- Design limitation: A single-decoder residual-modeling alternative was difficult to train because the bottom level reconstructed almost perfectly early, while auxiliary objectives caused serious top-level distortions.This experiment motivated the reported multi-scale design choice.
- Music priors and upsamplers: The model uses autoregressive token prediction at each level, conditioning on artist and genre embeddings and, for upsamplers, tokens from the upper level.Upper-level tokens pass through a conditioner network using dilated convolutions and a transposed 1-D convolution before being added at higher temporal resolution.
- Music VQ-VAE: The encoder compresses raw audio into embeddings, and the bottleneck converts them into codebook vectors whose indices form the discrete representation modeled by the priors.Compression and codebook size trade off fidelity against coherence or modeling difficulty.
- Lyrics conditioning: When trained on lyrics, the top-level prior uses segment-aligned lyric data in an encoder-decoder Transformer with sparse factorized self-attention.Its attention patterns repeat row, column, and previous-row types, with encoder-decoder attention interleaved when present.
B.3. Hyperparameters
The hyperparameter section presents separate tables for the VQ-VAE, middle- and bottom-level upsamplers, and top-level prior, alongside Transformer and convolutional width specifications.
- Hyperparameters: Transformer MLP blocks use the model width, while attention queries, keys, and values use 0.25 times the model width.Convolutional residual blocks use the same number of channels as the model width.
- Hyperparameters: Table 4 is the VQ-VAE hyperparameters table.The supplied passage identifies the table but does not enumerate its parameter values.
- Hyperparameters: Table 5 is the middle- and bottom-level upsampler hyperparameters table.The supplied passage identifies the table but does not enumerate its parameter values.
- Hyperparameters: Table 6 is the top-level prior hyperparameters table.The supplied passage identifies the table but does not enumerate its parameter values.
- Embedding analysis: The artist-and-genre embedding visualization shows genre clusters and artist groupings in a t-SNE plot.Pop appears centrally among several genres, while soundtrack and classical form separate clusters.