PaperlayerView exact revision on arXiv

Source-linked AI summary

SoundStream: An End-to-End Neural Audio Codec

Neil Zeghidour, Alejandro Luebs, Ahmed Omran, Jan Skoglund, Marco Tagliasacchi

arXiv:2107.03312v1cs.SDcs.LGeess.AS

TL;DR

SoundStream addresses efficient compression of speech, music, and general audio at low-to-medium bitrates. It uses an end-to-end fully convolutional codec with residual vector quantization, trained with reconstruction and adversarial losses, and adds quantizer dropout for bitrate scalability. The model supports streamable real-time inference on a smartphone CPU and jointly performs compression and enhancement without additional latency.

  • Problem

    Existing neural audio codecs largely target speech at low bitrates, motivating a codec for diverse audio content with efficient compression.

    Source: §I · p. 1

  • Method

    SoundStream jointly trains a fully convolutional encoder, residual vector quantizer, and decoder with reconstruction and adversarial losses, using quantizer dropout for variable bitrates.

    Source: §I · pp. 1–2

  • Results

    SoundStream outperforms Opus and EVS over a wide range of bitrates, while learning the encoder improves ViSQOL from 3.33 to 3.96 at 6 kbps.

    Source: §I · p. 2, D. Ablation studies · p. 8

  • Takeaways & Limitations

    The codec combines diverse-content compression, bitrate scalability, streamable real-time inference, and joint compression and enhancement without additional latency.

    Source: §VI · pp. 10–11

Abstract

from arXiv · show

We present SoundStream, a novel neural audio codec that can efficiently compress speech, music and general audio at bitrates normally targeted by speech-tailored codecs. SoundStream relies on a model architecture composed by a fully convolutional encoder/decoder network and a residual vector quantizer, which are trained jointly end-to-end. Training leverages recent advances in text-to-speech and speech enhancement, which combine adversarial and reconstruction losses to allow the generation of high-quality audio content from quantized embeddings. By training with structured dropout applied to quantizer layers, a single model can operate across variable bitrates from 3kbps to 18kbps, with a negligible quality loss when compared with models trained at fixed bitrates. In addition, the model is amenable to a low latency implementation, which supports streamable inference and runs in real time on a smartphone CPU. In subjective evaluations using audio at 24kHz sampling rate, SoundStream at 3kbps outperforms Opus at 12kbps and approaches EVS at 9.6kbps. Moreover, we are able to perform joint compression and enhancement either at the encoder or at the decoder side with no additional latency, which we demonstrate through background noise suppression for speech.

I. INTRODUCTION

SoundStream is an end-to-end neural audio codec designed for efficient compression across speech, music, and general audio. Its learnable architecture combines convolutional encoding and decoding, residual vector quantization, adversarial and reconstruction training, and support for scalable, low-latency operation.

Contribution

  • SoundStream compresses speech, music, and general audio more efficiently than previous codecs while targeting low-to-medium bitrates.The codec is designed to deliver high perceptual quality across diverse audio content.Source: §I · p. 1

Architecture

  • A fully convolutional encoder maps time-domain waveforms to lower-rate embeddings, which a residual vector quantizer discretizes before convolutional decoding reconstructs audio.The encoder, quantizer, and decoder form the core codec architecture.Source: §I · p. 1, Figure 2 · p. 2

Training

  • End-to-end training combines reconstruction and adversarial losses to optimize the codec components jointly.Discriminators distinguish decoded from original audio and provide a feature-based reconstruction space.Source: §I · p. 2

Bitrate scalability

  • Quantizer dropout enables one SoundStream model to handle different bitrates without requiring separate bitrate-specific models.The technique is introduced as a way to support bitrate variability through residual vector quantization.Source: §I · p. 2

Evaluation

  • Subjective evaluations report that SoundStream outperforms Opus and EVS over a wide range of bitrates.The comparison is reported using subjective quality metrics.Source: §I · p. 2

Deployment

  • The codec supports streamable, low-latency inference and runs in real time on a single smartphone CPU thread.A variant also performs audio compression and enhancement jointly without additional latency.Source: §I · p. 2

II. RELATED WORK

SoundStream builds on neural audio coding, generative audio, enhancement, and vector quantization to target diverse audio at low-to-medium bitrates with low latency.

  • The model jointly performs audio enhancement and compression without additional latency, including controllable denoising through time-dependent conditioning.Source: §II · pp. 2–3
  • SoundStream extends learnable vector quantization with a residual multi-stage quantizer trained end-to-end with the rest of the model.Source: §II · p. 3
  • Neural audio codecs learn efficient audio representations from data rather than relying on handcrafted signal-processing components.Source: §II · p. 3
  • A single SoundStream model compresses speech, music, and general audio at 24 kHz across 3 kbps to 18 kbps in real time on a smartphone CPU.Source: §II · p. 3
  • SoundStream is reported to outperform Opus and EVS over a broad range of bitrates in subjective evaluations.Source: §II · pp. 2–3

III. MODEL

SoundStream comprises a fully convolutional encoder, a residual vector quantizer, and a decoder trained end-to-end with adversarial and reconstruction losses.

  • The model processes a single-channel waveform through an encoder, residual vector quantizer, and decoder.Source: §III · p. 3
  • The encoder maps the input waveform to a sequence of embeddings, while the quantizer compresses each embedding to a target bitrate.Source: §III · p. 3
  • The decoder reconstructs audio from quantized embeddings, and the model is trained jointly with a discriminator.Source: §III · p. 3

B. Decoder architecture

The decoder mirrors the encoder to reconstruct the waveform, while a jointly trained vector quantizer compresses embeddings and supports multiple target bitrates.

Decoder architecture

  • The decoder uses transposed convolutions for up-sampling followed by residual units, reversing the encoder’s strides to restore input resolution.Source: B. Decoder architecture · p. 4

Residual vector quantization

D. Discriminator architecture

SoundStream uses wave-based and STFT-based fully convolutional discriminators to compute adversarial losses over waveform and time-frequency representations.

  • The discriminator architecture includes a wave-based discriminator and an STFT-based discriminator operating on real and imaginary STFT components.Source: D. Discriminator architecture · p. 5

Wave-based discriminator

  • The wave-based discriminator applies three structurally identical models at original, 2-times down-sampled, and 4-times down-sampled resolutions.Source: D. Discriminator architecture · p. 5

STFT-based discriminator

E. Training objective

SoundStream is trained with a combined adversarial, feature, and multi-scale spectral reconstruction objective to balance perceptual quality and signal fidelity.

  • SoundStream’s generator G(x) decodes the quantized encoder representation into the reconstructed waveform ˆx.Source: E. Training objective · p. 5
  • The generator loss combines adversarial, feature, and multi-scale spectral reconstruction losses.The overall loss is a weighted sum of these components.Source: E. Training objective · pp. 5–6
  • Adversarial training promotes perceptual quality through discriminators operating on STFT and waveform representations.Source: E. Training objective · p. 6
  • The feature loss measures average absolute differences between discriminator-layer activations for generated and target audio.Source: E. Training objective · p. 6
  • The spectral reconstruction loss compares 64-bin mel-spectrogram frames across multiple window scales.Each mel-spectrogram uses a hop length equal to one quarter of its window length.Source: E. Training objective · p. 6

F. Joint compression and enhancement

SoundStream integrates compression and enhancement in one model, allowing denoising to be enabled or disabled at inference time without adding latency.

IV. EVALUATION SETUP

SoundStream is trained and evaluated across clean speech, noisy speech, music, and additional real-world reverberant speech with background noise at 24 kHz.

  • The training data comprise clean speech, noisy speech, and music, all sampled at 24 kHz.Source: A. Datasets · p. 6
  • Noisy speech is synthesized by mixing LibriTTS speech with Freesound noise at uniformly sampled gains from −30 dB to 0 dB.Source: A. Datasets · pp. 6–7
  • Music training uses the MagnaTagATune dataset, with randomly selected three-second crops peak-normalized before processing.Source: A. Datasets · p. 7
  • Evaluation uses disjoint test splits and an additional real-world dataset containing near-field and far-field reverberant speech, sometimes with background noise.Source: A. Datasets · p. 7

B. Evaluation metrics

The evaluation combines crowdsourced subjective listening tests with objective ViSQOL measurements and compares SoundStream against established low-bitrate codecs.

C. Bitrate scalability

SoundStream’s quantizer dropout enables one model to operate across bitrates with little quality loss, while encoder capacity and quantizer design expose quality–complexity trade-offs.

C. Bitrate scalability

  • Quantizer dropout closes the quality gap caused by evaluating an 18 kbps model at lower bitrates without matching training conditions.The quality drop increases as the difference between training and inference bitrate grows.Source: C. Bitrate scalability · p. 8
  • A bitrate-scalable model trained with quantizer dropout matches bitrate-specific quality at 6 kbps and 12 kbps, with only a slight loss at 3 kbps.At 9 kbps and 12 kbps, the scalable model marginally outperforms bitrate-specific models.Source: C. Bitrate scalability · p. 8

D. Ablation studies

  • A learnable encoder improves ViSQOL from 3.33 to 3.96 at 6 kbps compared with a fixed mel-filterbank encoder.Learning the encoder also achieves ViSQOL 3.76 at 3 kbps, exceeding the fixed-encoder result.Source: D. Ablation studies · p. 8
  • The default 32-channel encoder and decoder run in real time on one Pixel 4 CPU thread at RTF > 2.3×, while 16 channels increase RTF beyond 7.1× with marginal quality loss.Using a smaller encoder raises encoder RTF to 18.6× while ViSQOL drops only from 3.96 to 3.94.Source: D. Ablation studies · p. 9
  • At 6 kbps, fewer residual quantizers with larger codebooks improve coding efficiency but increase computational complexity.A sequence of 80 one-bit quantizers causes only modest quality degradation.Source: D. Ablation studies · p. 9
  • Three 6 kbps configurations with different latency levels have equivalent audio quality, while increasing latency significantly increases computational cost.The per-frame budget rises from 80 to 160 bits when latency doubles.Source: D. Ablation studies · p. 10

E. Joint compression and enhancement

SoundStream can jointly compress audio and suppress background noise by conditioning either before quantization or at decoding, while denoising before quantization yields greater bitrate savings.

E. Joint compression and enhancement

  • SoundStream evaluates joint compression and background-noise suppression with conditioning applied either at the encoder side or decoder side.Models are trained at different bitrates for both configurations.Source: E. Joint compression and enhancement · p. 10
  • Encoder-side denoising and fixed denoising provide substantial bitrate savings compared with decoder-side denoising.The bitrate lower bound is estimated using cross-entropy between training and test quantization-symbol distributions.Source: E. Joint compression and enhancement · p. 10

F. Joint vs. disjoint compression and enhancement

The paper compares joint compression and enhancement with separate denoising and compression pipelines, evaluating quality on unseen noisy speech across multiple input noise levels.

F. Joint vs. disjoint compression and enhancement

  • SoundStream’s joint denoiser-codec is compared with SEANet denoising followed by SoundStream compression.The comparison uses models evaluated on VCTK noisy-speech clips at four input signal-to-noise ratios.Source: F. Joint vs. disjoint compression and… · p. 10
  • The joint model is evaluated on 1000 two-second VCTK clips for each of 0, 5, 10, and 15 dB input signal-to-noise ratios.VCTK was not used to train SoundStream or SEANet.Source: F. Joint vs. disjoint compression and… · p. 10

VI. CONCLUSIONS

  • SoundStream is presented as a neural codec that outperforms state-of-the-art codecs across a wide range of bitrates and content types.Its encoder, residual vector quantizer, and decoder are trained end-to-end with adversarial and reconstruction losses.Source: §VI · p. 10
  • The codec supports streamable real-time inference on a single smartphone CPU and combines compression and enhancement without additional latency.Quantizer dropout provides bitrate scalability with minimal performance loss versus bitrate-specific models.Source: §VI · p. 11

Related papers

Loading 2107.03312v1…