Source-linked AI summary
SoundStream: An End-to-End Neural Audio Codec
Neil Zeghidour, Alejandro Luebs, Ahmed Omran, Jan Skoglund, Marco Tagliasacchi
TL;DR
SoundStream addresses efficient compression of speech, music, and general audio at low-to-medium bitrates. It uses an end-to-end fully convolutional codec with residual vector quantization, trained with reconstruction and adversarial losses, and adds quantizer dropout for bitrate scalability. The model supports streamable real-time inference on a smartphone CPU and jointly performs compression and enhancement without additional latency.
Problem
Existing neural audio codecs largely target speech at low bitrates, motivating a codec for diverse audio content with efficient compression.
Method
SoundStream jointly trains a fully convolutional encoder, residual vector quantizer, and decoder with reconstruction and adversarial losses, using quantizer dropout for variable bitrates.
Results
SoundStream outperforms Opus and EVS over a wide range of bitrates, while learning the encoder improves ViSQOL from 3.33 to 3.96 at 6 kbps.
Takeaways & Limitations
The codec combines diverse-content compression, bitrate scalability, streamable real-time inference, and joint compression and enhancement without additional latency.
Abstract
from arXiv · showhide
We present SoundStream, a novel neural audio codec that can efficiently compress speech, music and general audio at bitrates normally targeted by speech-tailored codecs. SoundStream relies on a model architecture composed by a fully convolutional encoder/decoder network and a residual vector quantizer, which are trained jointly end-to-end. Training leverages recent advances in text-to-speech and speech enhancement, which combine adversarial and reconstruction losses to allow the generation of high-quality audio content from quantized embeddings. By training with structured dropout applied to quantizer layers, a single model can operate across variable bitrates from 3kbps to 18kbps, with a negligible quality loss when compared with models trained at fixed bitrates. In addition, the model is amenable to a low latency implementation, which supports streamable inference and runs in real time on a smartphone CPU. In subjective evaluations using audio at 24kHz sampling rate, SoundStream at 3kbps outperforms Opus at 12kbps and approaches EVS at 9.6kbps. Moreover, we are able to perform joint compression and enhancement either at the encoder or at the decoder side with no additional latency, which we demonstrate through background noise suppression for speech.
I. INTRODUCTION
SoundStream is an end-to-end neural audio codec designed for efficient compression across speech, music, and general audio. Its learnable architecture combines convolutional encoding and decoding, residual vector quantization, adversarial and reconstruction training, and support for scalable, low-latency operation.
- Contribution: SoundStream compresses speech, music, and general audio more efficiently than previous codecs while targeting low-to-medium bitrates.The codec is designed to deliver high perceptual quality across diverse audio content.
- Architecture: A fully convolutional encoder maps time-domain waveforms to lower-rate embeddings, which a residual vector quantizer discretizes before convolutional decoding reconstructs audio.The encoder, quantizer, and decoder form the core codec architecture.
- Training: End-to-end training combines reconstruction and adversarial losses to optimize the codec components jointly.Discriminators distinguish decoded from original audio and provide a feature-based reconstruction space.
- Bitrate scalability: Quantizer dropout enables one SoundStream model to handle different bitrates without requiring separate bitrate-specific models.The technique is introduced as a way to support bitrate variability through residual vector quantization.
- Evaluation: Subjective evaluations report that SoundStream outperforms Opus and EVS over a wide range of bitrates.The comparison is reported using subjective quality metrics.
- Deployment: The codec supports streamable, low-latency inference and runs in real time on a single smartphone CPU thread.A variant also performs audio compression and enhancement jointly without additional latency.
II. RELATED WORK
SoundStream builds on neural audio coding, generative audio, enhancement, and vector quantization to target diverse audio at low-to-medium bitrates with low latency.
- The model jointly performs audio enhancement and compression without additional latency, including controllable denoising through time-dependent conditioning.
- SoundStream extends learnable vector quantization with a residual multi-stage quantizer trained end-to-end with the rest of the model.
- Neural audio codecs learn efficient audio representations from data rather than relying on handcrafted signal-processing components.
- A single SoundStream model compresses speech, music, and general audio at 24 kHz across 3 kbps to 18 kbps in real time on a smartphone CPU.
- SoundStream is reported to outperform Opus and EVS over a broad range of bitrates in subjective evaluations.
III. MODEL
SoundStream comprises a fully convolutional encoder, a residual vector quantizer, and a decoder trained end-to-end with adversarial and reconstruction losses.
- The model processes a single-channel waveform through an encoder, residual vector quantizer, and decoder.
- The encoder maps the input waveform to a sequence of embeddings, while the quantizer compresses each embedding to a target bitrate.
- The decoder reconstructs audio from quantized embeddings, and the model is trained jointly with a discriminator.
B. Decoder architecture
The decoder mirrors the encoder to reconstruct the waveform, while a jointly trained vector quantizer compresses embeddings and supports multiple target bitrates.
- Decoder architecture: The decoder uses transposed convolutions for up-sampling followed by residual units, reversing the encoder’s strides to restore input resolution.
- Residual vector quantization: The quantizer compresses encoder embeddings to a target bitrate expressed in bits per second.
- Residual vector quantization: A vector quantizer maps each D-dimensional embedding frame to a codebook entry and represents it with S log2 N bits.
- Residual vector quantization: Codebook initialization with first-batch k-means centroids and replacement of unused vectors improve codebook usage.
- Residual vector quantization: Quantizer dropout samples a random number of active quantizers during training, enabling one model to operate across corresponding bitrates.
D. Discriminator architecture
SoundStream uses wave-based and STFT-based fully convolutional discriminators to compute adversarial losses over waveform and time-frequency representations.
- The discriminator architecture includes a wave-based discriminator and an STFT-based discriminator operating on real and imaginary STFT components.
- Wave-based discriminator: The wave-based discriminator applies three structurally identical models at original, 2-times down-sampled, and 4-times down-sampled resolutions.
- STFT-based discriminator: The STFT-based discriminator computes a single-scale STFT and processes it with a 2D convolution followed by residual blocks.
- STFT-based discriminator: Six residual blocks alternate time-frequency strides, progressively increasing channels before aggregating logits across down-sampled frequency bins.
E. Training objective
SoundStream is trained with a combined adversarial, feature, and multi-scale spectral reconstruction objective to balance perceptual quality and signal fidelity.
- SoundStream’s generator G(x) decodes the quantized encoder representation into the reconstructed waveform ˆx.
- The generator loss combines adversarial, feature, and multi-scale spectral reconstruction losses.The overall loss is a weighted sum of these components.
- Adversarial training promotes perceptual quality through discriminators operating on STFT and waveform representations.
- The feature loss measures average absolute differences between discriminator-layer activations for generated and target audio.
- The spectral reconstruction loss compares 64-bin mel-spectrogram frames across multiple window scales.Each mel-spectrogram uses a hop length equal to one quarter of its window length.
F. Joint compression and enhancement
SoundStream integrates compression and enhancement in one model, allowing denoising to be enabled or disabled at inference time without adding latency.
- Traditional pipelines place enhancement before compression or after decoding, so separate processing steps can add end-to-end latency.
- SoundStream combines compression with background-noise suppression using a conditioning signal that selects denoising modes during inference.
- FiLM layers transform network features using coefficients derived from a two-dimensional one-hot denoising-mode encoding.The conditioning can vary over time, adjusting denoising strength during processing.
- Applying conditioning at the bottleneck on either the encoder or decoder side was effective, with no further improvements from deeper placements.
IV. EVALUATION SETUP
SoundStream is trained and evaluated across clean speech, noisy speech, music, and additional real-world reverberant speech with background noise at 24 kHz.
- The training data comprise clean speech, noisy speech, and music, all sampled at 24 kHz.
- Noisy speech is synthesized by mixing LibriTTS speech with Freesound noise at uniformly sampled gains from −30 dB to 0 dB.
- Music training uses the MagnaTagATune dataset, with randomly selected three-second crops peak-normalized before processing.
- Evaluation uses disjoint test splits and an additional real-world dataset containing near-field and far-field reverberant speech, sometimes with background noise.
B. Evaluation metrics
The evaluation combines crowdsourced subjective listening tests with objective ViSQOL measurements and compares SoundStream against established low-bitrate codecs.
- 200 evaluation samples spanning clean, noisy, and reverberant speech plus music were each rated 20 times by screened native English-speaking headphone users.The subjective protocol was inspired by MUSHRA and used a hidden reference without a lowpass-filtered anchor.
- ViSQOL was selected as a freely available objective metric for development and hyperparameter selection.PESQ and POLQA were considered inconvenient because of licensing restrictions.
- SoundStream was compared with Opus, EVS, and Lyra across multiple bitrates in subjective evaluations.
- 3 kbps SoundStream significantly outperformed Opus at 6 kbps and EVS at 5.9 kbps, while matching quality required at least 12 kbps Opus or 9.6 kbps EVS.
- SoundStream quality remained consistent between clean and noisy speech, while music was more challenging because of its content diversity.
C. Bitrate scalability
SoundStream’s quantizer dropout enables one model to operate across bitrates with little quality loss, while encoder capacity and quantizer design expose quality–complexity trade-offs.
- C. Bitrate scalability: Quantizer dropout closes the quality gap caused by evaluating an 18 kbps model at lower bitrates without matching training conditions.The quality drop increases as the difference between training and inference bitrate grows.
- C. Bitrate scalability: A bitrate-scalable model trained with quantizer dropout matches bitrate-specific quality at 6 kbps and 12 kbps, with only a slight loss at 3 kbps.At 9 kbps and 12 kbps, the scalable model marginally outperforms bitrate-specific models.
- D. Ablation studies: A learnable encoder improves ViSQOL from 3.33 to 3.96 at 6 kbps compared with a fixed mel-filterbank encoder.Learning the encoder also achieves ViSQOL 3.76 at 3 kbps, exceeding the fixed-encoder result.
- D. Ablation studies: The default 32-channel encoder and decoder run in real time on one Pixel 4 CPU thread at RTF > 2.3×, while 16 channels increase RTF beyond 7.1× with marginal quality loss.Using a smaller encoder raises encoder RTF to 18.6× while ViSQOL drops only from 3.96 to 3.94.
- D. Ablation studies: At 6 kbps, fewer residual quantizers with larger codebooks improve coding efficiency but increase computational complexity.A sequence of 80 one-bit quantizers causes only modest quality degradation.
- D. Ablation studies: Three 6 kbps configurations with different latency levels have equivalent audio quality, while increasing latency significantly increases computational cost.The per-frame budget rises from 80 to 160 bits when latency doubles.
E. Joint compression and enhancement
SoundStream can jointly compress audio and suppress background noise by conditioning either before quantization or at decoding, while denoising before quantization yields greater bitrate savings.
- E. Joint compression and enhancement: SoundStream evaluates joint compression and background-noise suppression with conditioning applied either at the encoder side or decoder side.Models are trained at different bitrates for both configurations.
- E. Joint compression and enhancement: Encoder-side denoising and fixed denoising provide substantial bitrate savings compared with decoder-side denoising.The bitrate lower bound is estimated using cross-entropy between training and test quantization-symbol distributions.
F. Joint vs. disjoint compression and enhancement
The paper compares joint compression and enhancement with separate denoising and compression pipelines, evaluating quality on unseen noisy speech across multiple input noise levels.
- F. Joint vs. disjoint compression and enhancement: SoundStream’s joint denoiser-codec is compared with SEANet denoising followed by SoundStream compression.The comparison uses models evaluated on VCTK noisy-speech clips at four input signal-to-noise ratios.
- F. Joint vs. disjoint compression and enhancement: The joint model is evaluated on 1000 two-second VCTK clips for each of 0, 5, 10, and 15 dB input signal-to-noise ratios.VCTK was not used to train SoundStream or SEANet.
- VI. CONCLUSIONS: SoundStream is presented as a neural codec that outperforms state-of-the-art codecs across a wide range of bitrates and content types.Its encoder, residual vector quantizer, and decoder are trained end-to-end with adversarial and reconstruction losses.
- VI. CONCLUSIONS: The codec supports streamable real-time inference on a single smartphone CPU and combines compression and enhancement without additional latency.Quantizer dropout provides bitrate scalability with minimal performance loss versus bitrate-specific models.