Long-form music generation with latent diffusion

1. INTRODUCTION

• Prior text-conditioned models commonly train on 10–30s segments, and generating longer pieces from short-segment training can yield only local coherence.The authors identify 285s as a target that covers a high percentage of popular music and is within reach of current architectures and modern GPU VRAM.Source: §1 · p. 1
• The approach generates an entire piece at once without semantic tokens, using three stages rather than the four-stage pipeline typical of semantic-token approaches.The paper notes that single-stage text-to-waveform generation may be challenging because of GPU VRAM limitations.Source: §1 · p. 1
• The autoencoder’s continuous latent operates at 21.5Hz, compared with 40–150Hz in cited prior work, to support long temporal contexts without semantic tokens.The authors argue that maintaining perceptual quality at a low latent rate can be essential for training on long contexts.Source: §1 · p. 2
• The model generates coherent music over 285s (4m 45s), with state-of-the-art audio quality and text-prompt coherence, in 13s on a GPU.The authors report that it produces long-form music with coherent structure.Source: §1 · p. 2

2. LATENT DIFFUSION ARCHITECTURE

• The three components are a waveform autoencoder, a CLAP-based text-audio embedding model, and a transformer-based diffusion model operating in the autoencoder’s latent space.The model generates variable-length stereo music at 44.1kHz from text prompts, up to 4m 45s.Source: §2 · p. 2

2.1 Autoencoder

• The raw-waveform autoencoder uses strided convolutions for encoder downsampling and decoder upsampling, with a variational-autoencoder bottleneck and reconstruction, adversarial, and KL losses.Its Snake activation has a trainable β parameter, and the decoder omits the tanh activation used in DAC because the authors found it introduced harmonic distortion.Source: §2.1 · p. 2

2.2 Diffusion-transformer (DiT)

• The diffusion-transformer uses stacked attention and gated MLP blocks, with cross-attention for conditioning; block-wise attention and gradient checkpointing reduce long-sequence memory costs.Text and timing enter through cross-attention, while timing and the diffusion timestep are also prepended to the transformer.Source: §2.2 · p. 3

2.3 Variable-length music generation

• For variable-length generation, the model fills a specified window with music up to the requested length and silence afterward, which can be trimmed from the output.The model is trained to fill the rest of the signal with silence.Source: §2.3 · p. 3

2.4 CLAP text encoder

• The CLAP text encoder uses a RoBERTa-based text encoder and an HTSAT-based audio encoder with fusion, trained from scratch on the dataset with a language-audio contrastive loss.Text features come from the encoder’s next-to-last hidden layer.Source: §2.4 · p. 3

3. TRAINING SETUP

• The diffusion model is pre-trained on sequences up to 3m 10s for 70k GPU hours, then fine-tuned on sequences up to 4m 45s for 15k GPU hours.The autoencoder and CLAP model are trained first; their training requires approximately 16k and 3k GPU hours, respectively.Source: §3 · p. 3
• The DiT predicts a noise increment from noised ground-truth latents using the v-objective, and sampling uses 100 DPM-Solver++ steps with classifier-free guidance at scale 7.0.Models are trained with AdamW at a base learning rate of 1e−5 and exponential ramp-up and decay.Source: §3 · p. 3

3.1 Training data and prompt preparation

• The dataset contains 806,284 files totaling 19,500 hours, with music, sound effects, and instrument stems; the 285s target context covers over 90% of it.Prompts are formed from random subsets of metadata, with variations in metadata labels, delimiters, property order, and letter case.Source: §3.1 · p. 3

4.1 Quantitative evaluation

• The proposed model scores better than MusicGen at all evaluated lengths while being significantly faster, with no metric degradation after fine-tuning from 3m 10s to 4m 45s.The benchmark uses 586 no-singing prompts and evaluates 2-minute generations as well as each model’s maximum generation length.Source: §4.1 · p. 4

4.2 Qualitative evaluation

• Generations are comparable to ground truth in most aspects and superior to the baseline; the model earns good (4) MOS scores across the board, while 2m generations have fair (3) musical structure.Stereo correctness scores higher than 95%; the authors note high standard deviations in subjective structure ratings.Source: §4.2 · p. 4

4.3 Autoencoder evaluation

• The autoencoder’s raw reconstruction quality is comparable or marginally worse than available baselines, despite 2x–5x greater temporal downsampling and a lower latent rate.The comparison includes STFT distance, MEL distance, and SI-SDR; results against discrete codecs are not strictly comparable.Source: §4.3 · pp. 4–5

4.4 Musical structure analysis

• SSMs show generated music can repeat early motifs in later sections, while MusicGen examples rarely repeat early sections or can loop on middle or ending sections.Generated middle sections can also be repetitive while retaining an intro and outro; Figure 4 columns show rock, pop, jazz, hip-hop, and classical.Source: §4.4 · p. 6, Figure 4 · p. 6

4.5 Memorization analysis

• After ranking generations by LAION-CLAP similarity to training audio, the authors listened to the 50 closest candidates and found no memorization.The audit includes generations from 5,566 prompts tied to repeated training data and 586 out-of-distribution prompts.Source: §4.5 · p. 6

4.6 Additional creative capabilities

• Audio-initialized sampling supports style transfer while maintaining reference structure; vocal prompts produce vocal-like melodies without intelligible words, and shorter training examples support sound effects and instrument samples.These additional capabilities are briefly described and showcased, not thoroughly evaluated.Source: §4.6 · p. 6

• The model produces coherent music with state-of-the-art results over a 4m 45s target context.Source: §5 · p. 6
• An autoencoder compresses full musical tracks substantially in the temporal dimension before generation in its latent space.Source: §5 · p. 6
• A diffusion approach using a diffusion-transformer models the full-track latent representations.Source: §5 · p. 6

• The authors report no instances of memorization in their study using their methodology.Source: §6 · p. 7
• The technology is intended to support human music production and provide artists and content creators with variable-length, long-form stereo music from text.Source: §6 · p. 7
• The authors identify training-data bias as a concern and stress careful contextual consideration and collaboration with stakeholders, including artists and data providers.Source: §6 · p. 7