PaperlayerView exact revision on arXiv

Source-linked AI summary

Stable Audio 3

Zach Evans, Julian D. Parker, Matthew Rice, CJ Carr, Zack Zukowski, Josiah Taylor, Jordi Pons

arXiv:2605.17991v1cs.SDcs.AI

TL;DR

Diffusion models often generate entire audio durations at once, making short outputs inefficient and limiting practical deployment. Stable Audio 3 combines variable-length latent diffusion, inpainting, and fast post-trained generation, producing up to 6m 20s of stereo audio in under 2s on an H200 while achieving state-of-the-art results among open-weight systems.

  • Problem

    Diffusion models typically generate entire audio durations at once, making short outputs computationally and memory inefficient.

  • Method

    Stable Audio 3 combines a semantic-acoustic autoencoder with a flow-matching diffusion transformer, variable-length generation, inpainting, distillation warmup, and adversarial post-training.

  • Results

    Under 2s of inference produces up to 6m 20s of stereo audio, while Stable Audio 3 achieves state-of-the-art results for instrumental music and sound effects among open-weight systems.

  • Takeaways & Limitations

    Stable Audio 3 supports efficient variable-length generation and inpainting-based editing across music and sound effects, including on consumer-grade hardware.

  • Takeaways & Limitations

    Inference uses a length-independent logSNR-uniform schedule despite length-dependent timestep shifting during training, creating a train–inference mismatch.

Abstract

from arXiv · show

Stable Audio 3 is a family of fast latent diffusion models (small, medium, large) for variable-length audio generation and editing. Since our models can generate several minutes of audio, variable-length generations are key to avoid the cost of producing full-length generations for short sounds. We also support inpainting, enabling targeted audio editing and the continuation of short recordings. Our latent diffusion models operate on top of a novel semantic-acoustic autoencoder that projects audio into a compact latent space, enabling efficient diffusion-based generation while preserving audio fidelity and encouraging semantic structure in the latent. Finally, we run adversarial post-training to both accelerate inference and improve generation quality, reducing the number of inference steps while improving fidelity and prompt adherence. Stable Audio 3 models are trained on licensed and Creative Commons data to generate music and sounds in less than a 2s on an H200 GPU and less than a few seconds on a MacBook Pro M4. We release the weights of small and medium, that can run on consumer-grade hardware, together with their training and inference pipeline.

1 Introduction

Stable Audio 3 introduces variable-length latent diffusion models with inpainting, a semantic-acoustic autoencoder, and adversarial post-training for efficient, controllable audio generation. The released small and medium models are trained on licensed and Creative Commons data and support consumer-grade hardware.

  • Variable-length generation: Variable-length generation allocates L embeddings proportional to requested duration d, avoiding fixed-length computation on silence for short clips.This addresses the inefficiency of generating an entire predefined maximum length when only a short sample is requested.
  • Audio editing: Inpainting supports single-segment edits, multi-segment edits, and coherent continuation beyond an audio recording’s original endpoint.Users mask target segments for editing while preserving the unmasked original audio.
  • Semantic-acoustic representation: The semantic-acoustic autoencoder uses a compact latent space with 4096× downsampling designed to preserve high-fidelity reconstruction and semantic structure while remaining tractable for diffusion.The latent representation is intended to support generative modeling and downstream semantic use.
  • Fast inference: Adversarial post-training reduces diffusion sampling steps while maintaining or improving output quality, targeting responsive creative-tool inference.The training pipeline consists of three stages, including flow matching and adversarial post-training.
  • Accessibility and performance: Stable Audio 3 releases small and medium weights for consumer-grade hardware and generates up to 6m 20s in less than 2s on an H200.The models are trained on licensed and Creative Commons data and can scale from datacenter GPUs to a MacBook Pro.

2 Architecture

Stable Audio 3 combines a semantic-acoustic SAME autoencoder with a diffusion transformer that generates latent audio sequences. The architecture supports text conditioning, duration control, variable-length generation, and inpainting while compressing 44.1 kHz stereo audio into a compact latent representation.

  • System overview: The system combines a semantic-acoustic autoencoder mapping waveforms to continuous latents with a diffusion transformer conditioned on text, duration, and inpainting masks.The autoencoder and diffusion transformer form the two core architectural components.
  • SAME autoencoder: The SAME autoencoder combines 256-sample patching with Transformer Resampling Blocks for an additional 16× downsampling.Its training losses target reconstruction fidelity, generative tractability, and semantic structure through spectral, adversarial, diffusion-alignment, semantic-regression, and contrastive-latent objectives.
  • SAME autoencoder: 4096× compression produces 256-dimensional latent embeddings at approximately 10.76 Hz from 44.1 kHz stereo input.The soft-normalisation bottleneck uses a learnable affine transform and running standard deviation tracking to provide deterministic encoding.
  • Diffusion transformer: 64 learned memory embeddings provide a global context buffer before D transformer blocks process the projected SAME latents.The latent sequence is projected from 256 channels to the transformer dimensionality and projected back after memory embeddings are discarded.
  • Conditioning and capabilities: Conditioning uses AdaLN for diffusion timestep and duration, cross-attention for T5Gemma text and duration embeddings, and local-additive conditioning for inpainting.Stable Audio 3 also incorporates differential attention, memory embeddings, and variable-length support for text-to-audio generation and editing.

3 Training

Stable Audio 3 trains variable-length latent diffusion models through flow matching, one-step distillation, and adversarial post-training. The training design supports generation and inpainting while improving inference efficiency and preserving perceptual quality.

  • Multi-stage training: The multi-stage pipeline trains a flow-matching base model, then applies distillation warmup and adversarial post-training to improve speed and sample quality.All stages operate on pre-encoded SAME latents and use variable-length training.
  • Adversarial post-training: Adversarial post-training compares one-step predictions with real data, shifting the objective from conditional-mean regression toward true-distribution sampling and recovering perceptual sharpness.This stage discards the teacher, allowing the student to surpass the teacher’s quality.
  • Variable-length training: Variable-length training right-pads latent sequences, masks padding from attention and loss, and adapts the noise timestep per sample according to unpadded length.Longer sequences are shifted toward higher noise levels because fixed schedules can under-noise them relative to shorter sequences.
  • Variable-length training: Silence augmentation randomly extends signals with silence embeddings, averaging 4 sec of extension, to decouple duration conditioning from signal length and improve termination robustness.The silence latent is obtained by encoding a zero-valued waveform.
  • Generation and inpainting: All models are trained jointly for generation and inpainting using randomly sampled binary masks covering unconditional generation, random segments, or causal prefixes.The full-mask unconditional case occurs with probability 80%, while random-segment masks occur with probability 10%.
  • Distillation warmup: Distillation trains the student to map any intermediate teacher-trajectory state xt directly to the endpoint x̂0 in one step, replacing multi-step ODE solving.The student uses the same architecture as the teacher and is trained with an MSE loss.

4 Inference

Inference uses adversarial post-training with ping-pong sampling to generate audio efficiently while correcting difficult denoising steps. The pipeline uses an 8-step logSNR-uniform schedule, supports variable-length generation, and does not require classifier-free guidance at inference.

  • Sampling: Adversarial post-training enables single-pass clean-output prediction, while ping-pong sampling decomposes the large denoising step into multiple smaller steps.Each iteration alternates denoising with stochastic re-noising at a lower timestep.
  • Sampling: Ping-pong sampling is self-correcting because re-noising can correct inaccurate early estimates instead of propagating them through subsequent steps.The method contrasts with ODE solvers, where errors can displace later integration from the best trajectory.
  • Timestep schedule: 8 sampling steps use timesteps equally spaced in logSNR over [−6.2, 2.0], providing a favorable trade-off between inference efficiency and generation quality.The schedule is recovered from logSNR values using t_i = σ(−λ_i).
  • Timestep schedule: Inference uses the same logSNR-uniform schedule regardless of requested duration, despite a train–inference mismatch caused by length-dependent timestep shifts during training.Training emphasizes broad timestep coverage, whereas inference emphasizes placement across only 8 steps.
  • Variable-length generation: Variable-length generation allocates latent sequences from the requested duration plus 6 s of silence padding, retaining only the effective embeddings corresponding to target audio content.The allocation uses fs=44,100 Hz and an autoencoder downsampling ratio of r=4,096.
  • Guidance: The model requires no classifier-free guidance at inference because distillation warmup internalizes CFG-enhanced teacher trajectories, with adversarial post-training further refining text alignment.Avoiding CFG removes the standard requirement for conditional and unconditional forward passes at each denoising step.

5 Discussion · 5.1 Methodology

Stable Audio 3 is evaluated as a fast, consumer-oriented family for instrumental music and sound-effects generation and editing, using licensed or Creative Commons data. The methodology compares diverse baselines with objective metrics, listening tests, curated datasets, and standardized variable-length and inpainting evaluations.

  • 5 Discussion: Stable Audio 3 targets fast instrumental music and sound-effects generation and editing on consumer-grade hardware, using licensed or Creative Commons data.The family is designed for fast operation and supports both generation and editing.
  • 5 Discussion: The evaluation covers state-of-the-art generation, fast inference, variable-length generation, inpainting, adversarial post-training, and consumer-hardware deployment.The discussion identifies these as the principal results examined in subsequent sections.
  • 5.1 Methodology: Models are compared against open-weight and internal systems spanning diffusion and autoregressive architectures using objective metrics and a subjective listening test.Baselines include Stable Audio 2.5, Stable Audio Open variants, and other systems, with comparisons adjusted for model capabilities.
  • 5.1 Methodology: Evaluation uses FAD, CLAP score, and inference time to measure distributional similarity, text-audio alignment, and standardized wall-clock latency.FAD is lower-is-better, CLAP is higher-is-better, and inference time is generally measured on an H200.
  • 5.1 Methodology: The study evaluates instrumental music on 424 Song Describer Dataset music-caption pairs and sound effects on BBC Sound Effects subsets capped at 120s, 30s, 10s, and 5s.The BBC subsets contain 10,491, 5,406, 1,537, and 393 audio-caption pairs, respectively.
  • 5.1 Methodology: A 14-participant Mean Opinion Score listening test rates overall quality, text relevance, and musicality.The criteria assess production quality and realism, prompt matching, and coherent melodies and harmonies.
  • 5.1 Methodology: Inpainting is tested on 120s music and 30s–120s sound recordings across single inpainting, double inpainting, and continuation settings.Masked regions span 2%–20% of audio with a 1s minimum; double regions are separated by at least 6s, while continuation preserves an initial 5s-to-20%-of-audio segment.
  • 5.1 Methodology: Inpainting quality is assessed with full-audio and inpaint-region FAD plus CLAP measures for prompt alignment and similarity to the original region.Identical randomly sampled regions are used across models; double-inpainting regions are evaluated independently, while full-audio FAD is computed once per audio.

5.2 Instrumental music generation

Instrumental music generation is evaluated on the SDD at 120- and 190-second durations. Stable Audio 3 medium and large improve musicality, while small remains competitive with open-weight baselines despite weaker performance than larger models.

  • Evaluation setup: The SDD evaluation uses 120-second generations for all applicable models and 190-second generations for longer-form comparison, excluding small from the latter.The 120-second setting matches small’s maximum length, while 190 seconds matches Stable Audio 2.5’s maximum generation length.
  • Results: Stable Audio models achieve the strongest overall performance across instrumental music generation metrics, with Stable Audio 2.5 remaining the best-performing baseline.The results are reported in Tables 3 and 4 for the 120- and 190-second settings.
  • Results: Stable Audio 3 medium and large substantially improve musicality, whereas the small variant performs noticeably worse than the larger models.The reported contrast is between the larger Stable Audio 3 variants and small across the instrumental music generation results.
  • Results: Despite its reduced size and CPU-optimized lightweight autoencoder, small remains competitive with open-weight baselines.Its competitiveness is reported alongside its weaker performance relative to the larger models.

5.3 Sound effects generation

Stable Audio 3 sound-effects generation is evaluated with 5-second targets, enabling comparison with recent open-weight systems. The models outperform baselines while medium and large use a single model trained for both instrumental music and sound effects.

  • Evaluation setup: 5-second target durations enable comparison with a broad set of recent open-weight sound-effects generation systems.Longer evaluations up to 120 seconds are discussed in Section 5.5.
  • Results: Stable Audio 3 models consistently outperform all baselines, with large and medium achieving the best overall performance.Woosh Flow remains a strong baseline despite a discrepancy between its FAD and OVL scores.
  • Results: Woosh Flow’s competitive subjective OVL quality is often penalized by FAD because it produces band-limited signals.This highlights that FAD and OVL can assess Woosh Flow differently.
  • Model specialization: Medium and large are trained for both instrumental music and sound-effects generation with a single model while achieving state-of-the-art performance in both domains.Because of small’s limited parameter budget, training instead uses two specialized models: small-music and small-sfx.

5.4 Variable-length instrumental music generation

Stable Audio 3 is designed for native variable-length instrumental music generation, making short-form inference efficient while supporting durations from 20s to 380s when possible. Performance is generally strongest at intermediate lengths and degrades at very short and very long durations.

  • Variable-length evaluation: 20s–380s: Stable Audio models are evaluated across multiple generation durations, while small-music supports up to 120s and Stable Audio 2.5 up to 190s.The evaluation covers 20s clips through 380s full-length generations when possible.
  • Variable-length evaluation: Stable Audio 2.5 generates shorter clips inefficiently because inference runs over the full 190s sequence, with most computation spent generating silence.Its fixed-length training uses 190s sequences, padding shorter examples with silence.
  • Native variable-length generation: Stable Audio 3 models are explicitly designed for native variable-length generation, with inference cost scaling naturally with output duration.This design enables efficient short-form generation.
  • Generation quality across durations: 120–190s: Stable Audio 3 models typically achieve their best performance at intermediate lengths, while 20s outputs show degraded FAD and CLAP.The short-duration degradation is attributed to a mismatch between loop-heavy short training samples and full-song evaluation data.
  • Generation quality across durations: 380s: Stable Audio 3 performance also degrades at very long durations, particularly on the reported metrics.The passage indicates stronger degradation at very long lengths, but the supplied text truncates before specifying which metric is most affected.

5.5 Variable-length sound effects generation

Table 8 evaluates sound-effects quality and inference speed across output durations and model variants, restricting comparisons to 120 seconds despite support up to 380 seconds. The proposed models consistently outperform baselines with fast inference, while FAD improves monotonically as generation length increases.

  • Evaluation setup: Table 8 evaluates sound-effects quality and inference speed across varying output durations and models.Although medium and large models support generation up to 380s, evaluation is restricted to a maximum duration of 120s for simpler comparison with existing baselines.
  • Results: The proposed models consistently outperform baselines across evaluated durations while maintaining fast inference times.
  • Results: FAD improves monotonically as generation length increases across all proposed variants.The paper hypothesizes that longer samples are predominantly field recordings and ambient soundscapes with lower acoustic diversity and slower temporal variation, yielding lower distributional discrepancy.

5.6 Audio editing capabilities

The section evaluates Stable Audio 3 editing through single- and double-region inpainting and continuation for music and sound effects. Small underperforms medium and large in FAD and produces less smooth transitions, attributed to reduced model capacity and a smaller CPU-optimized autoencoder.

  • Evaluation scope: The evaluation covers single- and double-region inpainting and continuation for both music and sound effects.Results are reported in Tables 9 and 10, respectively.
  • Model comparison: Small obtains worse FAD results than medium and large across both music and sound-effects editing.The passage attributes this to small’s reduced model capacity and smaller CPU-optimized autoencoder.
  • Model comparison: Informal listening indicates that small produces less smooth transitions during editing.This qualitative difference is reported alongside the FAD comparison.

5.7 Adversarial Post-Training discussion

Adversarial post-training compares favorably with pre-trained flow-matching models by reducing inference latency and generation steps while improving generation quality. The comparison covers music and sound-effects models across Tables 11 and 12.

  • Inference efficiency: Post-trained models generate faster and support inference with as few as a single sampling step, unlike base models requiring 50 steps.The passage attributes the speedup to distillation warmup and adversarial post-training.
  • Quality and latency: Base models require 50 sampling steps, causing substantially higher latency and inferior generation quality than post-trained models.The comparison is between pre-trained flow-matching models and models further trained with distillation warmup and adversarial post-training.
  • Evaluation coverage: Table 11 compares pre-trained and post-trained music models at various sampling steps.The table provides the music-model comparison referenced in the discussion.
  • Evaluation coverage: Table 12 compares pre-trained and post-trained sound-effects models at various sampling steps.The table provides the sound-effects-model comparison referenced in the discussion.

5.8 VRAM Memory Usage · 5.9 Inference Times Across Hardware Platforms

Stable Audio 3’s peak VRAM usage rises with model size and generation duration, while the small model remains suitable for consumer hardware. Inference-time comparisons use fixed eight-step sampling across MacBook Pro M4 and NVIDIA H200 configurations, with platform-specific acceleration.

  • 5.8 VRAM Memory Usage: Peak VRAM usage increases with both model size and generation duration, with small having the lowest memory footprint.The comparison covers different models and generation durations.
  • 5.8 VRAM Memory Usage: Below 2.5 GB, small’s peak VRAM remains under this threshold even for 120-second generations.This is the reported long-duration result for small.
  • 5.8 VRAM Memory Usage: Approximately 6.5 GB and 9.0 GB, respectively, are required by medium and large at longer generation durations.These values describe peak memory usage for the two larger models.
  • 5.8 VRAM Memory Usage: Consumer-grade compatibility extends to small on RTX 3050-class GPUs and medium on GPUs including the RTX 3060 and RTX 4060.small supports generation lengths up to 120 seconds; medium requires approximately 6.5 GB for long-form generation.
  • 5.9 Inference Times Across Hardware Platforms: Inference-time experiments use a fixed configuration of 8 ping-pong sampling steps across four MacBook Pro M4 and NVIDIA H200 execution settings.The settings include CPU-only M4, CoreML-accelerated M4, standard PyTorch H200, and TensorRT-accelerated H200.
  • 5.9 Inference Times Across Hardware Platforms: The study reports end-to-end generation latency across different generation durations and model scales for these hardware and acceleration configurations.The supplied passage describes the comparison scope but does not provide the latency values.
  • 5.9 Inference Times Across Hardware Platforms: CPU-only MacBook Pro M4 results accelerate small with CoreML and SAME-S with TFLite, while medium and large decoders use PyTorch torch.compile.TensorRT cannot accelerate SAME-L’s sliding-window attention, whereas torch.compile can better exploit it.

6 Conclusion

Stable Audio 3 is a family of fast latent diffusion models for instrumental music and sound-effects generation and editing. Its semantic-acoustic autoencoder and flow-matching diffusion transformer support efficient, high-quality audio generation with few inference steps.

  • Stable Audio 3 comprises small, medium, and large latent diffusion models for instrumental music and sound-effects generation and editing.
  • The models combine a semantic-acoustic autoencoder with 4096× downsampling and a diffusion transformer trained using flow matching, distillation warmup, and adversarial post-training.
  • 8 ping-pong sampling steps produce up to 6m 20s of stereo audio at 44.1 kHz in under 2s on an H200 GPU.
Loading 2605.17991v1…