Source-linked AI summary
AudioDec: An Open-source Streaming High-fidelity Neural Audio Codec
Yi-Chiao Wu, Israel D. Gebru, Dejan Marković, Alexander Richard
TL;DR
Live audio codecs must balance bitrate, latency, and reconstruction quality. AudioDec addresses this with an open-source, modular, streamable end-to-end neural codec and an efficient training paradigm. It provides high-fidelity 48 kHz audio and real-time operation, while serving as a practical benchmark for codec applications.
Problem
Existing codecs and end-to-end neural codecs leave gaps in reconstruction quality, open-source benchmarking, efficient training, application flexibility, and comparisons with vocoder-based methods.
Method
AudioDec combines a modular end-to-end neural codec, an efficient training paradigm, HiFi-GAN-based vocoding, and group convolutions for real-time CPU operation.
Results
AudioDec delivers high-fidelity 48 kHz speech and real-time streamability, with a 25 ms window length reported on both GPU and CPU.
Takeaways & Limitations
AudioDec is presented as a high-quality, efficient, convenient, open-source benchmark for audio codec research and applications.
Abstract
from arXiv · showhide
A good audio codec for live applications such as telecommunication is characterized by three key properties: (1) compression, i.e.\ the bitrate that is required to transmit the signal should be as low as possible; (2) latency, i.e.\ encoding and decoding the signal needs to be fast enough to enable communication without or with only minimal noticeable delay; and (3) reconstruction quality of the signal. In this work, we propose an open-source, streamable, and real-time neural audio codec that achieves strong performance along all three axes: it can reconstruct highly natural sounding 48~kHz speech signals while operating at only 12~kbps and running with less than 6~ms (GPU)/10~ms (CPU) latency. An efficient training paradigm is also demonstrated for developing such neural audio codecs for real-world scenarios. Both objective and subjective evaluations using the VCTK corpus are provided. To sum up, AudioDec is a well-developed plug-and-play benchmark for audio codec applications.
1. INTRODUCTION
Audio codecs reduce audio signals to transmissible codes, but conventional and hybrid approaches face quality gaps, ad hoc designs, or limited flexibility. AudioDec addresses these issues with an open-source, modular, streamable end-to-end neural codec and an efficient training paradigm.
- Motivation: Audio codecs compress audio into lower-bitrate codes and reconstruct signals for transmission or storage.They support applications including secure communication, mobile and internet communications, and live media streaming.
- Limitations of Existing Codecs: Conventional parametric codecs maintain low bitrates but often sacrifice quality because of bandwidth limitations and limited modeling capacity.Modern versions improve reconstruction at 16 kHz or higher, yet a gap remains between natural and reconstructed audio.
- Neural Codec Background: Hybrid neural codecs improve selected modules but still depend on ad hoc designs for speech signals.Neural encoder-decoder structures have also been applied to handcrafted acoustic features such as spectrograms, phonological features, and pitches.
- Research Gap: End-to-end raw-waveform autoencoder codecs achieve high-fidelity speech generation but lack an open-source benchmark, clear efficient training paradigm, vocoder comparisons, and application flexibility.These limitations motivate a more practical codec system.
- AudioDec: AudioDec provides an open-source, modular end-to-end codec with replaceable encoders and decoders, HiFi-GAN integration, group convolutions, and objective and subjective evaluations.The paper reports high-fidelity 48 kHz speech generation and practical real-time operation across GPU and CPU settings.
2. BASELINE NEURAL AUDIO CODEC
The SoundStream-based baseline uses an end-to-end autoencoder with residual vector quantization and causal convolutions for low-bitrate streaming. Its training combines metric, adversarial, feature-matching, and codebook losses, but discriminator-heavy training and scenario-specific models limit efficiency and flexibility.
- Codec Architecture: An end-to-end autoencoder encodes raw waveforms into lower-temporal-resolution representations, quantizes them into transmission codes, and decodes those codes back into waveforms.The typical module sequence is encoder, projector, quantizer, and decoder.
- SoundStream Codec: SoundStream is an end-to-end autoencoder codec using residual vector quantization and fully causal convolutions for continuous streaming.A single model supports bitrates from 3 kbps to 18 kbps, and SoundStream is competitive with Opus at 12 kbps and EVS at 9.6 kbps using 3 kbps.
- Training Objectives: SoundStream training combines mel spectral metric loss with adversarial, feature-matching, and EMA codebook losses.The mel loss compares input and reconstructed mel spectrograms, while discriminators operate on complex spectrograms or waveforms.
- Baseline Limitations: GAN training is time-consuming because high-fidelity generation usually requires multiple deep discriminators, although metric losses can model low-frequency components alone.The adversarial stage mainly improves waveform details, high-frequency components, and phase synchronization.
- Baseline Limitations: Training models from scratch with metric and adversarial losses is inefficient, and separate SoundStream models are required for different application scenarios.This limits practical flexibility, including adjustments such as switching between mono and binaural outputs.
3. PROPOSED NEURAL AUDIO CODEC
AudioDec combines efficient staged training, modular encoder–decoder development, and a HiFi-GAN-based discriminator design for flexible, high-quality streaming neural audio coding.
- 3. PROPOSED NEURAL AUDIO CODEC: AudioDec uses an efficient training paradigm, modularized architecture, HiFi-GAN-based multi-period discrimination, and group convolution for real-time CPU execution.These design choices target training efficiency, model flexibility, audio quality, and real-time operation.
- 3. PROPOSED NEURAL AUDIO CODEC: The codec emphasizes phase information and a powerful decoder so low-bitrate codes retain essential information while reconstructing high-fidelity waveforms.The design separates essential encoded content from detailed waveform reconstruction.
- 3.1. Efficient Training Paradigm: 200k iterations train the generator with the initial objective, followed by 500k iterations training the whole model with updated mel spectral and adversarial losses.The second stage updates both the mel spectral objective and adversarial objectives; least squares GAN training improves stability.
- 3.2. Modularized Architecture: Fixing a pretrained quantizer and complete codebook enables rapidly developing interchangeable encoders and decoders for scenarios such as denoising and binaural rendering.The standard quantizer and codebook are obtained from a high-quality clean corpus before scenario-specific modules are developed.
- 3.2. Modularized Architecture: Symmetric encoder–decoder architectures stabilize autoencoder training, while asymmetric designs are unstable and symmetric powerful designs are inefficient.The encoder must preserve limited essential information, whereas the decoder must reconstruct high-resolution waveform details.
- 3.3. HiFi-GAN-based Multi-Period Discriminator: Replacing the redundant STFT discriminator with multi-period discriminators improves audio quality by capturing periodic details across different segment lengths.Multi-period discriminators target periodic detail, complementing long-term dependency modeling associated with STFT and multi-scale discriminators.
4. EXPERIMENTS
Experiments evaluate AudioDec on 48 kHz speech using objective and subjective quality measures, training speed, and streaming latency. Vocoder-based AudioDec variants achieve the strongest quality results, while the proposed training paradigm improves development efficiency and several models remain potentially streamable.
- 4.2. Objective Evaluation: The vocoder-based AudioDec codecs achieve the best performance across all objective measurements at 48 kHz and 12.8 kbps.The comparison includes MCD, LSD, DNSMOS SIG, F0RMSE, and U/V errors.
- 4.3. Subjective Evaluation: Vocoder-based AudioDec codecs significantly outperform baseline SoundStream in subjective quality, although both remain below natural speech.AD v1 achieves the strongest perceptual result, while AD v1 and AD v0 show that group convolutions perform competitively with MRF networks.
- 4.4. Discussion: Training with metric loss alone is much faster than training with discriminators, and fixing the encoder during decoder and discriminator training slightly improves speed.These results support using the proposed training paradigm to develop encoders for different applications.
- 4.4. Discussion: 3.5 hrs is sufficient to train an encoder for a new application such as denoising on an A100 GPU.The reported training time improves the efficiency of developing codecs for different scenarios.
- 4.4. Discussion: SymAD, AD v1, and AD v2 are potentially streamable with 12.5 ms buffers on a GPU, while SymAD and AD v2 retain this potential on a four-thread CPU.A standalone pipeline worked with a 25 ms GPU buffer using AD v1 and a 35 ms CPU buffer using AD v2.
5. CONCLUSION
AudioDec is an open-source, high-quality, efficient, and convenient benchmark for real-time audio codec research. Its modular architecture and training paradigm support flexible development across scenarios.
- AudioDec provides an open-source benchmark for high-fidelity 48 kHz audio codec research.
- Its modularized architecture improves speech quality through a powerful vocoder and supports codec development for different scenarios.
- The efficient training paradigm reduces training time for new encoders while achieving better quality.
- Low-latency implementations make AudioDec streamable in real time with a 25 ms window on both GPU and CPU.