Source-linked AI summary
High Fidelity Neural Audio Compression
Alexandre Défossez, Jade Copet, Gabriel Synnaeve, Yossi Adi
TL;DR
Audio compression must reduce bitrate while preserving perceptual quality, but neural codecs must represent diverse signals and compress efficiently. EnCodec addresses these challenges with a streaming neural architecture and training improvements, achieving state-of-the-art quality across evaluated speech and music settings while a Transformer further reduces bandwidth by up to 40%.
Problem
Audio compression seeks lower bitrate and perceptual distortion, while neural models must represent diverse signals without artifacts and compress efficiently in compute time and size.
Method
EnCodec uses a streaming convolutional encoder-decoder with sequential latent modeling, spectrogram adversarial loss, gradient balancing, and optional Transformer compression.
Results
EnCodec achieves state-of-the-art speech and music scores from 1.5 to 24 kbps across 24 kHz monophonic and 48 kHz stereophonic audio, while Transformer compression reduces bandwidth by up to 40%.
Takeaways & Limitations
The model provides high-fidelity real-time neural audio compression across sample rates and bandwidths, with additional bandwidth savings for applications where low latency is not essential.
Takeaways & Limitations
At 48 kHz, the increased number of processing steps leads to slower-than-real-time processing, although more efficient implementations or accelerated hardware may improve it.
Abstract
from arXiv · showhide
We introduce a state-of-the-art real-time, high-fidelity, audio codec leveraging neural networks. It consists in a streaming encoder-decoder architecture with quantized latent space trained in an end-to-end fashion. We simplify and speed-up the training by using a single multiscale spectrogram adversary that efficiently reduces artifacts and produce high-quality samples. We introduce a novel loss balancer mechanism to stabilize training: the weight of a loss now defines the fraction of the overall gradient it should represent, thus decoupling the choice of this hyper-parameter from the typical scale of the loss. Finally, we study how lightweight Transformer models can be used to further compress the obtained representation by up to 40%, while staying faster than real time. We provide a detailed description of the key design choices of the proposed model including: training objective, architectural changes and a study of various perceptual loss functions. We present an extensive subjective evaluation (MUSHRA tests) together with an ablation study for a range of bandwidths and audio domains, including speech, noisy-reverberant speech, and music. Our approach is superior to the baselines methods across all evaluated settings, considering both 24 kHz monophonic and 48 kHz stereophonic audio. Code and models are available at github.com/facebookresearch/encodec.
1 Introduction
EnCodec addresses the need for efficient, perceptually faithful audio compression with a real-time neural encoder-decoder codec. It combines quantization, adversarial training, and human evaluation across speech and music.
- 82% of 2021 internet traffic was attributed to streaming audio and video, motivating more efficient audio compression.
- Neural lossy compression must represent diverse signals without producing artifacts while remaining efficient in compute time and model size.
- EnCodec uses an encoder-decoder architecture with residual vector quantization and reconstruction plus adversarial losses.
- The study evaluates intertwined architecture, quantization, and perceptual-loss choices using objective ablations and MUSHRA tests on speech and music.
2 Related Work
Prior work spans neural audio generation, traditional low-bitrate codecs, neural latent-space codecs, and discrete audio representations. EnCodec builds most directly on neural codecs using quantization and adversarial perceptual losses.
- Autoregressive neural audio generation produced convincing audio but incurred slow inference, while GAN-based approaches targeted natural-sounding quality.
- Traditional codecs such as Opus and EVS support general audio across bitrates, sampling rates, and real-time compression.
- Recent neural audio codecs commonly quantize latent representations before decoding them, using approaches including uniform quantization and Gumbel-Softmax.
- SoundStream is the closest related model, combining a fully convolutional encoder-decoder, residual vector quantization, reconstruction loss, and adversarial perceptual losses.
- Hierarchical VQ-VAE and self-supervised methods have also used discrete audio or speech representations for generation and representation learning.
3 Model
The model is a streaming convolutional encoder-decoder with quantized latent representations, optional Transformer entropy coding, and reconstruction and adversarial objectives. Training is stabilized with a gradient-based loss balancer.
- 3.1 Encoder & Decoder Architecture: EnCodec encodes audio into a latent representation, quantizes it with vector quantization, and decodes the compressed representation into reconstructed audio.
- 3.1 Encoder & Decoder Architecture: The streaming convolutional encoder-decoder applies sequential modeling over the latent representation on both encoder and decoder sides, supporting 24 kHz and 48 kHz audio.
- 3.1 Encoder & Decoder Architecture: The encoder uses convolutional residual blocks, strided downsampling, a two-layer LSTM, and a final convolution; the decoder reverses the downsampling with transposed convolutions.
- 3.1 Encoder & Decoder Architecture: The streamable setup places padding before the first time step and can output 320 samples, or 13 ms, after receiving the first 13 ms of audio.
- 3.2 Quantization: Residual vector quantization progressively quantizes residuals with additional codebooks, with variable codebook counts supporting multiple bandwidth targets.
- 3.3 Language Modeling and Entropy Coding: A small Transformer language model optionally entropy-codes quantized units to reduce bandwidth while targeting faster-than-real-time single-core compression and decompression.
- 3.4 Training objective: The training objective combines time- and frequency-domain reconstruction, perceptual adversarial losses, and residual-quantization commitment loss.
- 3.4 Training objective: The loss balancer stabilizes training by addressing differing discriminator-gradient scales and makes loss weights easier to interpret independently of loss magnitude.
4 Experiments and Results
Experiments evaluate EnCodec across diverse audio domains, codecs, bandwidths, perceptual objectives, streaming settings, and stereo audio. EnCodec consistently outperforms evaluated baselines while retaining strong streaming performance and substantial bandwidth savings from entropy coding.
- Experimental setup: The evaluation covers 24 kHz monophonic speech, noisy speech, music, and general audio, with testing across clean speech, mixed speech, and music categories.Training uses diverse datasets and on-the-fly mixtures, with reverberation added for some samples.
- Experimental setup: MUSHRA tests use hidden references and low anchors, with 50 five-second samples per category and at least 10 annotations per sample.Annotators rate perceptual quality from 1 to 100, with filtering of noisy annotations and outliers.
- Baseline comparison: At equal bandwidth, EnCodec achieves higher MUSHRA scores than all evaluated baselines, including Lyra-v2, Opus, and EVS.At 3 kbps, EnCodec performs better on average than Lyra-v2 at 6 kbps and Opus at 12 kbps.
- Baseline comparison: 25–40% bandwidth reductions are achieved by applying an additional language model over the codes, including reducing a 3 kbps model to 1.9 kbps.The compression ratio is lower at higher bandwidths.
- Ablation study: A single multi-scale STFT discriminator is sufficient for high-quality audio, simplifying training and reducing training time; adding MPD provides a small MUSHRA gain.The discriminator comparison uses ViSQOL, SI-SNR, and MUSHRA metrics.
- Ablation study: Streamable modeling causes a small degradation relative to non-streamable modeling, but performance remains strong while enabling streaming inference.Stereo evaluation also finds EnCodec significantly outperforms Opus at 6 kbps and is comparable to MP3 at 64 kbps.
5 Conclusion
EnCodec is presented as a real-time neural audio compression model delivering high-fidelity audio across sample rates and bandwidths. Its design combines adversarial training, gradient balancing, and Transformer-based bandwidth reduction.
- EnCodec is a state-of-the-art real-time neural audio compression model producing high-fidelity samples across sample rates and bandwidths.
- The model covers settings from 24 kHz monophonic audio at 1.5 kbps to 48 kHz stereophonic audio.
- A spectrogram-only adversarial loss reduces artifacts and improves sample quality.
- A gradient balancer stabilizes training and makes loss weights more interpretable.
- A small Transformer reduces bandwidth by up to 40% without further quality degradation when low latency is not essential.
A.1 Experimental details
The experimental details describe dataset construction and a reimplementation of SoundStream for comparative evaluation. Dataset splits are defined per source, while the baseline follows the original model’s reported implementation details.
- Common Voice uses 99.5% of samples for training, 0.25% for validation, and the remainder for testing.
- The SoundStream baseline was reimplemented because its original implementation was not open sourced.
- The reimplementation uses k-means initialization, exponential-moving-average updates, and random restarts for residual vector quantization.
A.2.1 DiffQ Quantizer
The DiffQ quantizer approximates non-differentiable scalar quantization during training with scaled additive noise, then uses normalized, clamped quantization at evaluation time. Its formulation also supports bandwidth control and sparse representations.
- DiffQ uses scaled additive uniform noise during training to simulate non-differentiable scalar quantization.
- The training representation clamps z to m − L·σ and m + L·σ before adding noise scaled by L·σ.
- A bandwidth penalty is added when the differentiable bandwidth estimate exceeds a target.
- At test time, z is normalized and clamped to [0, 1], with the number of quantization levels set to round(2^B).
- Scaling the noise relative to the signal as B approaches zero yields a sparse representation.
A.2.2 Gumbel softmax quantizer
The Gumbel-softmax quantizer uses multiple codebooks and differentiable approximately one-hot assignments during training. A learned prior supports arithmetic coding and bandwidth minimization.
- The quantizer contains NC codebooks, each with Ω entries, and represents assignments through differentiable approximately one-hot vectors.
- During testing, Gumbel-softmax is replaced by sampling from each codebook’s distribution.
- A learned prior distribution over codebook entries is used with arithmetic coding for the quantized representation.
- Cross-entropy between the prior and posterior jointly trains the prior and provides a differentiable bandwidth estimate.
A.3 Additional Results
Additional experiments compare EnCodec with SoundStream and examine how architectural choices and loss balancing affect quality, speed, and training stability. The results support trade-offs between reconstruction quality, inference speed, and objective tuning.
- Comparing to SoundStream: At 6.0kbps, both EnCodec and SoundStream significantly outperform Opus and EVS.The comparison uses the authors’ SoundStream reimplementation and reports results in Table A.2.
- Comparing to SoundStream: With RVQ as the latent quantizer, EnCodec is superior to the SoundStream model.Using DiffQ, EnCodec is only slightly better than the SoundStream implementation; the RVQ comparison favors EnCodec.
- The effect of the model architecture: Increasing model capacity marginally changes objective scores but substantially worsens the real-time factor.The analysis profiles streamable 24 kHz models at 6 kbps on a single CPU thread.
- The effect of the model architecture: LSTM sequence modeling improves SI-SNR and final reconstruction quality at the cost of a lower real-time factor.The comparison evaluates streamable models with different channel counts, residual units, and sequence-modeling choices.
- The effect of the model architecture: Using residual units has more impact on the real-time factor and slightly degrades SI-SNR compared with the LSTM-based version.The residual-unit variant uses three residual units with progressively increased dilation.
- The effect of the balancer: The loss balancer significantly stabilizes training and reduces effort spent tuning objective coefficients without degrading performance.These experiments use the DiffQ quantizer and the Jamando music dataset.
A.4 Societal impact
The paper frames efficient audio compression as increasingly important because streaming content consumes substantial internet traffic and low-bitrate codecs lose quality on non-speech audio. Its societal relevance is tied to reducing storage and network demands while supporting communication and media services.
- Motivation: Audio and video streams represented 82% of internet traffic in 2021, according to Cisco.The paper links this share to user-generated content, calls, and streaming services.
- Motivation: Compression reduces the storage requirements and network bandwidth needed to serve audio and video content.The paper also identifies wearable-device adoption as increasing the importance of efficient compression.
- Motivation: At very low bitrates such as 3 kbps, Opus and EVS audio quality deteriorates on non-speech audio.The paper describes this as an essential challenge for very-low-bitrate compression with high fidelity.
- Motivation: Very-low-bitrate codecs can enable communication and improve experiences, motivating high-fidelity compression in this regime.The passage states this consequence without limiting it to a particular audio domain.