PaperlayerView exact revision on arXiv

Source-linked AI summary

Fast Timing-Conditioned Latent Audio Diffusion

Zach Evans, CJ Carr, Josiah Taylor, Scott H. Hawley, Jordi Pons

arXiv:2402.04825v3cs.SDcs.LGeess.AS

TL;DR

Existing audio diffusion systems face high computational demands and limited support for naturally variable-length, long-form stereo audio. Stable Audio combines VAE-based latent diffusion with text and timing conditioning, generating up to 95 sec of 44.1kHz stereo audio in 8 sec and performing strongly across two public benchmarks. The approach expands generation and evaluation to structured music, stereo sound effects, and long-form full-band stereo signals, while training-data biases remain a limitation.

  • Problem

    Raw-signal diffusion is computationally demanding, while fixed-size outputs poorly match the variable durations of music and sound effects.

  • Method

    Stable Audio uses VAE-based latent diffusion conditioned on text prompts and timing embeddings to control generated content and length.

  • Results

    95 sec of stereo 44.1kHz audio can be rendered in 8 sec on an A100 GPU, while Stable Audio achieves top or near-top performance in two public benchmarks.

  • Takeaways & Limitations

    Stable Audio supports efficient variable-length long-form stereo music and sound generation, including structured music and stereo sound effects.

  • Takeaways & Limitations

    The technology may reflect biases in its training data, raising concerns for cultures underrepresented in that dataset.

Abstract

from arXiv · show

Generating long-form 44.1kHz stereo audio from text prompts can be computationally demanding. Further, most previous works do not tackle that music and sound effects naturally vary in their duration. Our research focuses on the efficient generation of long-form, variable-length stereo music and sounds at 44.1kHz using text prompts with a generative model. Stable Audio is based on latent diffusion, with its latent defined by a fully-convolutional variational autoencoder. It is conditioned on text prompts as well as timing embeddings, allowing for fine control over both the content and length of the generated music and sounds. Stable Audio is capable of rendering stereo signals of up to 95 sec at 44.1kHz in 8 sec on an A100 GPU. Despite its compute efficiency and fast inference, it is one of the best in two public text-to-music and -audio benchmarks and, differently from state-of-the-art models, can generate music with structure and stereo sounds.

1. Introduction

Stable Audio addresses the computational cost of raw-audio diffusion and the difficulty of generating naturally variable-length music and sound effects. It combines latent diffusion with text and timing conditioning, supporting efficient long-form stereo generation and evaluation beyond standard short-form metrics.

  • Latent diffusion reduces the computational demands of audio generation by operating on a heavily downsampled latent representation rather than raw signals.This representation also supports faster inference and long-form audio generation.
  • Fixed-size audio diffusion outputs make variable-length training and generation difficult, often producing musical excerpts that begin or end mid-phrase.Models trained on fixed chunks commonly crop or pad longer recordings to the training length.
  • Stable Audio uses text prompts and timing embeddings to control both the content and variable length of generated music and sound effects.Timing conditioning allows specified durations up to the training window length.
  • 95 sec of stereo audio at 44.1kHz can be rendered in 8 sec on an A100 GPU with 40GB VRAM.
  • The work introduces metrics and qualitative assessments for long-form full-band stereo audio, including plausibility, semantic correspondence, text adherence, musicality, stereo correctness, and musical structure.It reports state-of-the-art results on long-form stereo music and sound-effects generation and structured music generation.
  • Reproducibility materials, including model and metric code and demos, are available online.

2. Related Work

Prior audio generators trade off speed, duration flexibility, audio fidelity, or modality coverage, while evaluation has largely targeted short-form mono audio. Stable Audio uses latent diffusion and timing conditioning to generate variable-length 44.1kHz stereo music and sounds efficiently, alongside metrics suited to long-form stereo outputs.

  • Autoregressive models support long-form variable-length generation sequentially but remain slow at inference.
  • Previous latent-diffusion systems pioneered text-to-music and text-to-audio generation, but differ in whether they decode latents to waveforms or spectrograms.
  • Prior work rarely covered commercial music specifications for both music and sound effects, especially stereo sound synthesis.Stable Audio focuses on 44.1kHz stereo music and sounds from text prompts.
  • Earlier non-autoregressive and diffusion systems generally used shorter training windows, with prior reported limits of 20 sec, 30 sec, or 44 sec for music.
  • Stable Audio uses latent diffusion and timing conditioning to generate variable-length stereo signals up to 95 sec at 44.1kHz in 8 sec on an A100 GPU.
  • Common audio metrics target short-form mono signals at 16kHz, motivating new quantitative and qualitative measures for long-form full-band stereo generations.The paper evaluates musicality, stereo correctness, and musical structure in addition to audio quality and text alignment.
  • One model generates both music and sound from text prompts, rather than specializing in a single audio task.

3. Architecture

Stable Audio combines a fully convolutional VAE, text and timing conditioning, and a U-Net latent diffusion model to generate stereo audio with user-controlled duration. Its 95-second generation window supports variable-length outputs by trimming silence after timing-conditioned synthesis.

  • Variational autoencoder: A fully convolutional VAE compresses 44.1kHz stereo audio into an invertible, lossy latent representation for faster generation and training.The encoder downsamples by 1024 and produces 64-channel latents, yielding an overall compression ratio of 32.
  • Text conditioning: A CLAP text encoder supplies conditioning features containing information about relationships between words and audio.
  • Timing conditioning: Timing embeddings encode seconds_start and seconds_total, then join text features in U-Net cross-attention to control generated output length.During inference, seconds_start=0 and seconds_total=30 for a 95-second model produces 30 seconds of audio followed by 65 seconds of silence.
  • Diffusion model: The diffusion backbone is a 907M-parameter U-Net with symmetrical downsampling and upsampling blocks connected by skip connections.Prompt and timing information enter through cross-attention, while diffusion timestep conditioning modulates activations through FiLM layers.

4. Training

Stable Audio was trained on a large AudioSparx collection of music, sound effects, and stems with associated metadata. Training used large-scale mixed-precision optimization for the autoencoder, CLAP encoder, and diffusion model.

  • Dataset: 806,284 audios totaling 19,500 hours comprise the dataset, including music, sound effects, and instrument stems with text metadata.Music accounts for 66% of audios, sound effects 25%, and instrument stems 9%.
  • Autoencoder training: The autoencoder was trained for 1.1M steps on 16 A100 GPUs with effective batch size 256, freezing its encoder after 460,000 steps.The decoder was fine-tuned for the remaining 640,000 steps using stereo-aware multi-resolution sum-and-difference STFT losses.
  • Text encoder training: The CLAP model was trained from scratch for 100 epochs on 64 A100 GPUs with effective batch size 6,144 using language-audio contrastive loss.
  • Diffusion training: The diffusion model was trained for 640,000 steps on 64 A100 GPUs with effective batch size 256 using automatic mixed precision and exponential moving average.Audio was resampled to 44.1kHz and represented in 95.1-second windows, with cropping for longer files and end-padding for shorter files.
  • Prompt construction: Training prompts were formed by concatenating random subsets of natural-language and domain-specific metadata such as BPM, genre, moods, and instruments.

5. Methodology

The evaluation methodology adapts audio metrics and human assessments to long-form, full-band stereo generation. Quantitative experiments use MusicCaps and AudioCaps, with comparisons against representative open-source baselines and explicit output-length caveats.

  • Quantitative metrics: FDopenl3 evaluates plausibility by comparing generated and reference audio statistics in OpenL3 feature space for variable-length, full-band stereo signals.Left and right channels are embedded independently and concatenated; evaluation audio is resampled to 44.1kHz.
  • Quantitative metrics: KLpasst measures semantic correspondence through KL divergence between PaSST label probabilities for generated and reference audio.The metric is adapted for long-form audio rather than only short snippets.
  • Quantitative metrics: CLAPscore measures cosine similarity between prompt text and generated-audio embeddings using CLAPLAION feature fusion for inputs longer than 10 sec.The audio representation combines a global input with random crops at multiple time scales.
  • Human evaluation: The adapted quantitative metrics evaluate audio quality, text alignment, musicality, stereo correctness, and musical structure, while supporting variable-length inputs.Human ratings use mean opinion scores for quality, alignment, and musicality, with binary judgments for stereo correctness and structure.
  • Benchmarks and baselines: Experiments use 5,521 MusicCaps generations and 4,875 AudioCaps generations from original audio resampled to 44.1kHz.Baselines include AudioLDM2, MusicGen, and AudioGen, while direct comparison with models such as Moûsai and JEN1 is infeasible because their weights are unavailable.
  • Human evaluation: 19 users participated in the qualitative study, whose results report mean opinion scores with standard deviations and percentages.

6. Experiments

The experiments evaluate reconstruction, text encoders, timing control, benchmark quality, and inference speed. Stable Audio shows accurate length control and fast stereo audio generation relative to the compared systems.

  • Reconstruction: Stable Audio’s autoencoded training data yields slightly inferior FDopenl3 results than real training data for both MusicCaps and AudioCaps.The comparison assesses reconstruction quality in the latent space against the evaluation audio.
  • Text encoder: The text-encoder ablation compares CLAPLAION, CLAPours, and T5 embeddings using the base diffusion model on MusicCaps and AudioCaps.The encoders are evaluated after training the base diffusion model for 350k steps.
  • Timing conditioning: Timing conditioning is evaluated by comparing expected audio length with measured signal length, using silence detection to estimate the generated duration.A 30-second request is expected to produce 30 seconds of signal followed by silence within the 95-second window.
  • Benchmark comparison: Stable Audio outperforms the state of the art in audio quality and improves text alignment on MusicCaps, while remaining competitive in musicality and stereo correctness.The model also generates structured music, including intro, development, and outro, unlike models mainly capable of developing musical ideas.
  • Inference speed: Stable Audio is faster than autoregressive models and faster than AudioLDM2 variants despite operating at stereo 44.1kHz.Inference times are compared on one A100 GPU with batch size 1.

7. Conclusions

The paper presents a latent diffusion model for rapid, variable-length, long-form stereo audio generation at 44.1kHz. It reports strong benchmark performance together with structured music and stereo sound-effect generation.

  • Conclusion: Stable Audio enables rapid generation of variable-length, long-form stereo music and sounds at 44.1kHz from textual and timing inputs.The approach is reported as a latent diffusion model and is evaluated with new metrics for long-form full-band stereo signals.
  • Conclusion: Stable Audio is a top contender in two public benchmarks and can generate structured music and stereo sound effects unlike other state-of-the-art models.The conclusion describes both benchmark standing and capabilities beyond generation of musical ideas.

Impact Statement

The technology may assist audio production by generating variable-length, long-form stereo music and sound effects from text. The impact statement also identifies training-data bias and contextual concerns as risks requiring stakeholder collaboration.

  • Potential benefits: The technology offers variable-length, long-form stereo music and sound-effect generation from text descriptions for audio production tasks.The statement frames this capability as expanding artists’ and content creators’ tools.
  • Risks: Training-data bias may make the technology inappropriate for cultures underrepresented in the training dataset.The impact statement identifies this as a prominent concern.
  • Risks: The contextual nature of audio recordings and music motivates careful consideration and collaboration with artists, data providers, and other stakeholders.The authors commit to continued research and collaboration to navigate responsible AI-based audio production.
  • Inference trade-off: The quality-versus-inference-speed trade-off shows substantial quality improvement within the first 50 diffusion steps, motivating a 100-step setting.The passage notes diminishing returns with additional computational effort and leaves open more aggressive acceleration.

B. MusicCaps and AudioCaps: the original data from Youtube

The benchmark’s original YouTube data is predominantly stereo and full-band, so the evaluation uses versions resampled to 44.1kHz. Figures 5 and 6 summarize the original MusicCaps and AudioCaps data statistics.

  • Original data: MusicCaps and AudioCaps are commonly evaluated using mono 16kHz versions, although their original data is predominantly stereo and full-band.The paper uses the original data rather than the commonly used mono versions.
  • Original data: The original benchmark audio is resampled to 44.1kHz to match Stable Audio’s target bandwidth.This preserves evaluation of the original stereo full-band signals at the model’s target rate.
  • Dataset statistics: Figure 5 presents statistics of the original MusicCaps data, while Figure 6 presents statistics of the original AudioCaps data.The captions identify the datasets summarized by each figure.

C. Timing conditioning: additional evaluation

Stable Audio follows timing conditioning closely, generating audio near requested durations across 30-, 60-, and 90-second prompts. Lengths were measured by detecting signal silence with an energy threshold.

  • Timing conditioning generates signals of the specified length.The evaluation measures audio length by detecting when the signal becomes silence using a simple energy threshold.
  • 30, 60, and 90 sec prompts produce audios of the expected lengths with fairly precise timing control.Figure 7 groups measured lengths by specified duration: blue for 30 sec, red for 60 sec, and green for 90 sec.

D. Related work: additional discussion on latent diffusion models

The discussion contrasts Stable Audio with latent-diffusion systems in architecture, timing control, task scope, and benchmark performance. Stable Audio uses a fully-convolutional VAE and CLAP conditioning to target fast, variable-length synthesis of both music and sounds.

  • Comparison with Moûsai: Stable Audio uses a fully-convolutional end-to-end VAE, whereas Moûsai uses a spectrogram encoder and a 100-step diffusion decoder.The paper identifies this architectural distinction as crucial for achieving fast inference.
  • Comparison with JEN-1: Stable Audio targets both music and sound synthesis with one model, while Moûsai and JEN-1 target music synthesis.JEN-1 is trained for 10 sec music rather than variable-length, long-form music and sound effects.
  • Comparison with AudioLDM2: Stable Audio is faster than AudioLDM2 and outperforms it in audio quality and music text alignment, while AudioLDM2 leads on sound-effects text alignment.These comparisons are reported across Tables 1, 2, and 3.
  • Text conditioning: Stable Audio uses a CLAP text encoder trained on the same dataset as its diffusion model rather than the pretrained T5 or FLAN-T5 embeddings used by related systems.The authors state that this promotes consistency across model components by avoiding distribution or vocabulary mismatches.
  • Additional evaluation: On MusicCaps prompts without singing voice, Stable Audio consistently obtains better results than the other systems except for comparable MusicGen-large KLpasst scores.The comparison uses a subset of 2184 non-vocal prompts.

F. Implementation details

The implementation materials provide online code, configuration files, and resources for reproducing Stable Audio and its metrics. They also document the CLAP text-encoder training code used with a private dataset.

  • Code to reproduce Stable Audio is available online in the stable-audio-tools repository.The repository is provided as an implementation resource.
  • Configuration files for the VAE autoencoder and latent diffusion model are available online.The files define the training configurations and architecture settings for both components.
  • The configuration files provide compact descriptions of Stable Audio’s architecture and implementation.The authors describe them as additional resources for understanding the underlying implementation.
  • Code to reproduce the evaluation metrics is available online in the stable-audio-metrics repository.This resource supports reproduction of the reported metrics.
  • The CLAP authors’ code was used to train the text encoder with a private dataset.The implementation relies on code shared by the CLAP authors.
Loading 2402.04825v3…