Source-linked AI summary
SoundStorm: Efficient Parallel Audio Generation
Zalán Borsos, Matt Sharifi, Damien Vincent, Eugene Kharitonov, Neil Zeghidour, Marco Tagliasacchi
TL;DR
Long, high-quality audio generation is constrained by long codec-token sequences and costly autoregressive modeling. SoundStorm uses RVQ-aware bidirectional attention and confidence-based parallel decoding, achieving AudioLM-comparable quality and consistency two orders of magnitude faster while supporting natural multi-turn dialogue synthesis. The paper also notes that generated samples may reflect biases in the training data, including accents and voice characteristics.
Problem
Efficient generation of long, high-quality neural-audio-codec sequences remains an open problem because codec representations create long sequences and costly attention.
Method
SoundStorm uses an RVQ-aware architecture with bidirectional attention and coarse-to-fine confidence-based non-autoregressive decoding of SoundStream tokens.
Results
SoundStorm matches AudioLM’s quality with improved speaker and acoustic consistency, runs two orders of magnitude faster, and generates 30 seconds in 0.5 seconds on a TPU-v4.
Takeaways & Limitations
Combining SoundStorm with text-to-semantic modeling enables natural multi-turn dialogues with controllable content, speaker voices, and speaker turns.
Takeaways & Limitations
Generated audio may reflect biases in the training data, including represented accents and voice characteristics, and the paper leaves fuller analysis for future work.
Abstract
from arXiv · showhide
We present SoundStorm, a model for efficient, non-autoregressive audio generation. SoundStorm receives as input the semantic tokens of AudioLM, and relies on bidirectional attention and confidence-based parallel decoding to generate the tokens of a neural audio codec. Compared to the autoregressive generation approach of AudioLM, our model produces audio of the same quality and with higher consistency in voice and acoustic conditions, while being two orders of magnitude faster. SoundStorm generates 30 seconds of audio in 0.5 seconds on a TPU-v4. We demonstrate the ability of our model to scale audio generation to longer sequences by synthesizing high-quality, natural dialogue segments, given a transcript annotated with speaker turns and a short prompt with the speakers' voices.
1. Introduction
SoundStorm targets the open problem of efficiently generating long, high-quality neural-audio-codec sequences. It combines an RVQ-aware architecture with confidence-based parallel decoding and achieves AudioLM-compatible quality substantially faster.
- The challenge: Long, high-quality audio generation remains difficult because neural codecs require either exponentially large codebooks or long sequences with costly quadratic attention.The resulting trade-off is between perceptual quality and runtime.
- The challenge: RVQ creates a hierarchical token structure in which finer levels contribute less to perceptual quality, enabling specialized factorizations and decoding schemes.SoundStorm is motivated by exploiting this structure during training and inference.
- SoundStorm: SoundStorm combines an architecture adapted to hierarchical audio tokens with parallel, non-autoregressive, confidence-based decoding.The decoding scheme is inspired by MaskGIT and operates on residual-vector-quantized sequences.
- SoundStorm: The model sums embeddings within each SoundStream frame, uses bidirectional Conformer attention, and predicts tokens with separate heads for each RVQ level.This makes the self-attention sequence length depend on SoundStream frames rather than the number of quantizers.
- Results: SoundStorm replaces AudioLM’s coarse and fine acoustic stages while matching quality, improving speaker and acoustic consistency, and generating audio two orders of magnitude faster.It also supports high-quality natural dialogues with transcript-controlled content, speaker turns, and voice prompts.
2. Related work
Prior work addresses neural-audio-codec generation through hierarchical representations, RVQ-aware architectures, and parallel decoding. SoundStorm combines these directions to reduce the cost of modeling long multi-level token sequences.
- Neural audio codecs: Neural codecs enable high-quality audio reconstruction at low bitrates, supporting discrete token modeling for intelligible speech and broader audio generation.These representations extend discrete sequence modeling from speech embeddings to audio signals.
- Neural audio codecs: Flattened SoundStream token sequences impose quadratic self-attention cost as bitrate increases, limiting long-form high-quality audio generation.SoundStorm addresses this issue by modeling multi-level tokens with an RVQ-aware design.
- RVQ-aware architectures: RVQ-aware models commonly sum embeddings from the same frame and use separate heads or level-specific prediction strategies for different quantizers.AudioGen predicts RVQ levels in parallel, while VALL-E predicts the first level autoregressively and later levels non-autoregressively.
- RVQ-aware architectures: RQ-Transformer factorizes RVQ sequences with spatial and depth Transformers, but its combination with parallel decoding is described as a future avenue for audio generation.This work extends related RVQ modeling and parallel-decoding ideas to audio.
- Parallel decoding: MaskGIT iteratively predicts masked tokens using confidence scores and a schedule, progressively fixing more positions across decoding rounds.SoundStorm extends this parallel iterative sampling scheme to residual-quantized audio token sequences.
3. Method
SoundStorm converts time-aligned conditioning tokens into SoundStream tokens using frame-level aggregation, RVQ-specific prediction heads, and coarse-to-fine iterative parallel decoding. Its design exploits conditional independence among finer acoustic levels to reduce forward passes.
- Architecture: SoundStorm maps discrete conditioning tokens to SoundStream tokens that can be decoded into waveforms, assuming time alignment or compatible upsampling.Semantic tokens from AudioLM, SPEAR-TTS, and MusicLM are example conditioning signals.
- Architecture: The model interleaves conditioning and SoundStream tokens at each frame, sums their embeddings, and feeds the resulting frame-level sequence to a Conformer.This aggregation determines the bidirectional-attention sequence length by SoundStream frames, independent of RVQ level count.
- Masking and decoding: SoundStorm applies MaskGIT-style masking and confidence-based decoding independently across RVQ levels in coarse-to-fine order.This ordering respects dependencies between RVQ levels while exploiting conditional independence in finer levels.
- Masking and decoding: Training samples a timestep and RVQ level, masks selected tokens at that level together with all finer-level tokens, and computes loss only on masked tokens at the sampled level.The conditioning tokens and tokens before the sampled timestep remain unmasked to support voice prompting.
- Masking and decoding: Decoding begins with masked SoundStream tokens except any prompt, samples levels sequentially, and retains candidates according to confidence schedules within each level.The final iteration uses greedy decoding, which the authors found improved perceived audio quality.
- Masking and decoding: Finer RVQ tokens can be sampled in parallel because they represent local acoustic details under the conditional-independence assumption.Consequently, the number of forward passes can decrease as decoding progresses to finer levels.
4. Experiments
Experiments compare SoundStorm with AudioLM across speech quality, intelligibility, voice preservation, acoustic consistency, runtime, and decoding iterations. SoundStorm matches audio quality while improving several consistency and intelligibility measures, achieving substantial speedups with 16 first-level decoding iterations.
- Speech evaluation: SoundStorm matches AudioLM’s acoustic generator in audio quality while outperforming it in speech intelligibility and acoustic consistency.The comparison covers short, mid, and long LibriSpeech test-clean segments.
- Speech evaluation: SoundStorm significantly improves WER and CER over AudioLM on all segment-length splits in both prompted and unprompted settings.AudioLM uses slide-and-prompt generation for segments longer than 10 seconds because its autoregressive acoustic stage is prohibitively expensive for 30-second generation.
- Speech evaluation: SoundStorm significantly outperforms AudioLM in preserving the speaker identity of the prompt.Voice preservation is measured using cosine similarity between speaker embeddings from the prompt and generated audio.
- Speech evaluation: SoundStorm’s acoustic consistency remains close to the original samples, whereas AudioLM exhibits more drift over time, especially for longer audio.The analysis measures cosine similarity between embeddings from a 3-second prompt and subsequent 3-second crops.
- Runtime: 0.5 seconds is SoundStorm’s runtime for generating 30 seconds of audio on a TPU-v4, making it two orders of magnitude faster than AudioLM’s acoustic generator.Including semantic generation and SoundStream decoding, the full speech-continuation pipeline generates 30 seconds within 2 seconds.
- Ablations: 16 iterations in the first RVQ level increase the quality score by 0.1-0.2 over level-wise greedy decoding, while further iterations do not improve it.Experiments on LibriSpeech test-clean also found no statistically significant quality improvement from increasing iterations at RVQ levels 2-12.
5. Dialogue Synthesis
SoundStorm is used with text-to-semantic modeling to synthesize natural, multi-turn dialogues from transcripts and speaker prompts. The pipeline supports controlled speaker turns and produces 30-second segments efficiently.
- Dialogue synthesis targets maintaining speaker identities across multiple turns and long time spans.
- A corpus of approximately 100,000 hours of dialogue was segmented into 30-second chunks and transcribed with annotated speaker turns.
- The text-to-semantic stage uses a 1.2B-parameter ByT5-large Transformer to predict semantic tokens from byte-level text.
- Inference feeds transcript-derived semantic tokens into SoundStorm while using short speaker exchanges as prompts for both stages.
- 2 seconds produces a 30-second dialogue segment with high quality, natural disfluencies, and transcript-controlled speaker turns.
6. Conclusion
SoundStorm efficiently synthesizes high-quality audio from discrete conditioning tokens. Compared with AudioLM’s acoustic generator, it is faster, more temporally consistent over long samples, and supports natural multi-speaker dialogue from text.
- SoundStorm synthesizes high-quality audio from discrete conditioning tokens efficiently.
- Two orders of magnitude faster than AudioLM’s acoustic generator, SoundStorm achieves higher temporal consistency on long audio samples.
- Combining SoundStorm with text-to-semantic modeling enables longer-context text-to-speech with natural dialogues, multiple speaker turns, and control over voices and content.
7. Broader Impact
SoundStorm’s efficient audio generation and speaker control have broader implications for accessibility and misuse prevention. The paper identifies training-data limitations and voice-mimicry risks while describing detection safeguards and future watermarking work.
- SoundStorm’s samples may reflect biases in training data, including represented accents and voice characteristics.
- A more thorough analysis of the training data and its limitations remains future work.
- Voice mimicry can enable malicious applications such as bypassing biometric identification and impersonation.
- The authors verified that SoundStorm-generated audio remains detectable by a dedicated classifier and plan to explore audio watermarking.