Source-linked AI summary
HiFi-Codec: Group-residual Vector quantization for High Fidelity Audio Codec
Dongchao Yang, Songxiang Liu, Rongjie Huang, Jinchuan Tian, Chao Weng, Yuexian Zou
TL;DR
Audio codec models support audio generation, but high-quality reconstruction can require many codebooks, burdening generation models. The paper proposes GRVQ and HiFi-Codec, which achieve good reconstruction with four codebooks, while releasing AcademiCodec and author-reported training resources. The study also reports limitations in generalization validation, objective evaluation, and downstream-task coverage.
Problem
High-quality audio reconstruction can require many codebooks, increasing the burden on audio generation models, while codec training processes are not widely available.
Method
The paper proposes group-residual vector quantization and builds HiFi-Codec using an encoder, quantization layer, and decoder trained on public TTS datasets.
Results
HiFi-Codec achieves better reconstruction performance than Encodec while using 4 codebooks.
Takeaways & Limitations
AcademiCodec releases training processes and pretrained models for Encodec, SoundStream, and HiFi-Codec to facilitate audio codec and generation research.
Takeaways & Limitations
The study does not validate generalization on a large-scale universal dataset, include subjective evaluation, or provide enough downstream-task evaluations.
Abstract
from arXiv · showhide
Audio codec models are widely used in audio communication as a crucial technique for compressing audio into discrete representations. Nowadays, audio codec models are increasingly utilized in generation fields as intermediate representations. For instance, AudioLM is an audio generation model that uses the discrete representation of SoundStream as a training target, while VALL-E employs the Encodec model as an intermediate feature to aid TTS tasks. Despite their usefulness, two challenges persist: (1) training these audio codec models can be difficult due to the lack of publicly available training processes and the need for large-scale data and GPUs; (2) achieving good reconstruction performance requires many codebooks, which increases the burden on generation models. In this study, we propose a group-residual vector quantization (GRVQ) technique and use it to develop a novel \textbf{Hi}gh \textbf{Fi}delity Audio Codec model, HiFi-Codec, which only requires 4 codebooks. We train all the models using publicly available TTS data such as LibriTTS, VCTK, AISHELL, and more, with a total duration of over 1000 hours, using 8 GPUs. Our experimental results show that HiFi-Codec outperforms Encodec in terms of reconstruction performance despite requiring only 4 codebooks. To facilitate research in audio codec and generation, we introduce AcademiCodec, the first open-source audio codec toolkit that offers training codes and pre-trained models for Encodec, SoundStream, and HiFi-Codec. Code and pre-trained model can be found on: \href{https://github.com/yangdongchao/AcademiCodec}{https://github.com/yangdongchao/AcademiCodec}
1 Peking University, China 2 Tencent AI Lab, Shenzhen, China 3 Zhejiang University, China
The paper is indexed under vector quantization, audio codecs, and audio generation.
- The work concerns vector quantization for audio codec models.
- Audio codecs are a central application area.
- Audio generation is an intended use context.
1. INTRODUCTION
The introduction motivates audio codecs for generation tasks and identifies a tension between reconstruction quality and the number of codebooks. It proposes GRVQ and HiFi-Codec to address that tension.
- Audio codecs compress audio into discrete representations used by generation systems such as TTS and music generation.
- RVQ-based codecs use multiple codebooks, but later codebooks store sparse detail information after the first codebook captures most information.
- 12 codebooks are needed by Encodec for high-quality reconstruction, creating longer sequences that are harder for transformers to model.
- The paper targets good reconstruction with fewer codebooks for audio generation tasks.
- GRVQ splits latent features into groups, quantizes each with RVQ, and combines the group outputs.
- Splitting features into 2 groups with 2 residual layers produced good reconstruction performance compared with a pre-trained Encodec model.
2. RELATED WORKS
Related work establishes neural audio codecs as encoder–quantizer–decoder systems, with SoundStream and Encodec using RVQ. Prior approaches can require many codebooks, limiting suitability for generation tasks.
- SoundStream introduced RVQ for high-level representations carrying semantic information.
- Audio generation systems model discrete tokens produced by audio codecs, including AudioLM’s use of SoundStream representations.
- Neural audio codecs extract latent features with an encoder, quantize them, and pass them to a decoder for waveform reconstruction.
- The related codec designs use multiple VQ codebooks in residual quantization.
- Algorithm 1 describes group-residual quantization by splitting y into two groups, applying quantizers, and concatenating the outputs.
- Many codebooks can preserve reconstruction quality but may be unsuitable for generation tasks.
3. PROPOSED METHOD
HiFi-Codec combines an encoder, group-residual quantization layer, and decoder in an end-to-end audio reconstruction model. Its architecture draws on convolutional designs from Encodec and SoundStream.
- 3.1. Overview: HiFi-Codec contains an encoder E, quantization layer Q, and decoder G that reconstructs audio from compressed latent features.
- 3.1. Overview: The model is trained end-to-end with time-domain, frequency-domain, and multi-resolution discriminator perceptual losses.
- 3.1. Overview: Figure 1 provides a visual overview of the HiFi-Codec model.
- 3.1. Overview: The encoder and decoder use convolutional architectures inspired by Encodec and SoundStream.
3.3. Group-residual Vector Quantization (GRVQ)
GRVQ splits latent features into groups and applies separate residual vector quantizers, concentrating more codebooks in the first layer to reduce the total quantizer burden.
- 3.3. Group-residual Vector Quantization (GRVQ): GRVQ splits each latent feature representation into several groups, then applies multiple RVQs to quantize the groups separately.The study uses two groups, z1 and z2.
- 3.3. Group-residual Vector Quantization (GRVQ): The design addresses RVQ’s uneven information allocation, where the first codebook stores most information while later codebooks store less.
- 3.3. Group-residual Vector Quantization (GRVQ): The grouped quantized representations are combined to support waveform decoding with fewer residual quantization blocks.
3.4. Discriminator
The codec uses three discriminator types operating on spectrogram and waveform representations to provide adversarial perceptual supervision.
- 3.4. Discriminator: HiFi-Codec uses an MS-STFT discriminator, an MPD, and an MSD discriminator.The MS-STFT discriminator follows Encodec’s configuration, while MPD and MSD come from HiFi-GAN.
- 3.4. Discriminator: The MS-STFT discriminator processes multi-scaled complex-valued STFTs with real and imaginary parts concatenated.
3.5. Training Loss
Training jointly optimizes reconstruction, perceptual, adversarial, feature, and GRVQ commitment objectives for the generator and discriminators.
- 3.5. Training Loss: The GAN objective jointly optimizes the generator and discriminators using reconstruction, perceptual, and GRVQ commitment losses.
- 3.5. Training Loss: Reconstruction loss combines an L1 time-domain term with a multi-scale mel-spectrogram time-frequency term.
- 3.5. Training Loss: Adversarial supervision uses MS-STFT for spectrogram similarity and MPD/MSD for waveform similarity.
- 3.5. Training Loss: Feature loss averages the absolute differences between discriminator internal activations for generated and corresponding real audio.
- 3.5. Training Loss: The GRVQ commitment loss is computed for each group and residual quantizer.
- 3.5. Training Loss: The generator objective sums weighted adversarial, feature, reconstruction, and commitment losses: LossG = λadvLadv + λfeat · Lfeat + λrec · Lrec + λc · Lc.The λ terms balance the respective loss components during training.
4. EXPERIMENTS
Experiments evaluate reconstruction quality on public English and Chinese TTS data using PESQ and STOI, finding strong performance with a compact codebook configuration.
- 4. EXPERIMENTS: PESQ and STOI measure the gap between reconstructed and target audio.
- 4. EXPERIMENTS: Training uses public TTS datasets including LibriTTS, VCTK, and AISHELL, covering mainly English and Chinese speech.
- 4. EXPERIMENTS: HiFi-Codec achieves good reconstruction performance with only 4 codebooks.The best performance is reported with downsample times of 240 and 8 codebooks, using 4 codebooks per layer across two residual layers.
5. CONCLUSION
The study introduces GRVQ and HiFi-Codec for generation tasks, achieving better reconstruction performance than Encodec with only 4 codebooks. It also releases training processes for Encodec and SoundStream to support further codec research.
- HiFi-Codec uses group-residual vector quantization and is specially designed for generation tasks.
- HiFi-Codec achieves better reconstruction performance than Encodec while using 4 codebooks.
- The project releases training processes for Encodec and SoundStream so readers can train their own codec models.
- The authors plan to optimize HiFi-Codec and train better Encodec and SoundStream models.
6. LIMITATIONS
The authors identify limitations involving dataset scale, evaluation methodology, and downstream-task coverage. These constraints limit validation of generalization, subjective reconstruction quality, and generation-task performance.
- The models are not trained on a large-scale dataset, so generalization cannot be validated well.
- Objective evaluation metrics may not accurately assess reconstruction performance, while subjective evaluation is omitted.
- HiFi-Codec lacks enough downstream tasks to evaluate its performance for generation tasks.