Source-linked AI summary
High-Fidelity Audio Compression with Improved RVQGAN
Rithesh Kumar, Prem Seetharaman, Alejandro Luebs, Ishaan Kumar, Kundan Kumar
TL;DR
High-dimensional audio needs high-quality compression into discrete tokens for generative modeling, but existing codecs can produce artifacts and struggle across audio domains. The paper proposes Improved RVQGAN, combining universal convolutional audio compression with residual vector quantization and redesigned adversarial and perceptual losses. It reports ~90x compression of 44.1 KHz audio at 8 kbps, with stronger performance than competing methods and remaining difficulty on some challenging audio.
Problem
High-resolution audio is difficult to model and existing codecs can produce artifacts or specialize in limited audio domains.
Method
Improved RVQGAN combines convolutional encoder-decoder networks, residual vector quantization, periodic activations, projected codebook learning, and adversarial and perceptual losses.
Results
~90x compression of 44.1 KHz audio at 8 kbps is achieved, while the method outperforms competing audio compression algorithms.
Takeaways & Limitations
A single universal codec supports speech, music, environmental sounds, and other audio domains for high-fidelity generative audio modeling.
Takeaways & Limitations
The codec still has difficulty reconstructing some challenging audio despite outperforming competing approaches across domains.
Abstract
from arXiv · showhide
Language models have been successfully used to model natural signals, such as images, speech, and music. A key component of these models is a high quality neural compression model that can compress high-dimensional natural signals into lower dimensional discrete tokens. To that end, we introduce a high-fidelity universal neural audio compression algorithm that achieves ~90x compression of 44.1 KHz audio into tokens at just 8kbps bandwidth. We achieve this by combining advances in high-fidelity audio generation with better vector quantization techniques from the image domain, along with improved adversarial and reconstruction losses. We compress all domains (speech, environment, music, etc.) with a single universal model, making it widely applicable to generative modeling of all audio. We compare with competing audio compression algorithms, and find our method outperforms them significantly. We provide thorough ablations for every design choice, as well as open-source code and trained model weights. We hope our work can lay the foundation for the next generation of high-fidelity audio modeling.
1 Introduction
High-resolution audio generation requires compact representations that preserve fidelity across diverse audio types. The paper introduces Improved RVQGAN to address artifacts, limited compression, and domain-specific codecs.
- Motivation: ~44,100 samples per second and dependencies across multiple time scales make high-resolution audio generation difficult.Existing approaches often divide generation into audio synthesis and intermediate-representation prediction.
- Compression requirements: Discrete codes compress audio into a latent space while targeting high-fidelity reconstruction, strong compression, and broad audio coverage.The codec should discard imperceptible details while preserving high-level structure.
- Limitations of prior work: Existing codecs exhibit tonal, pitch, periodicity, and high-frequency artifacts and often specialize in speech or music rather than generic sounds.SoundStream and EnCodec only partially satisfy the desired codec properties.
- Contributions: Improved RVQGAN compresses 44.1 KHz audio into 8 kbps discrete codes at ~90x compression with minimal quality loss and fewer artifacts.The model is presented as a high-fidelity universal audio compression model.
- Contributions: The method addresses codebook collapse, quantizer-dropout quality loss, and codec design through improved codebook learning and adversarial and reconstruction components.The contributions include periodic inductive biases, multi-scale STFT discrimination, and multi-scale mel loss.
2 Related Work
Prior work established GAN-based high-fidelity audio synthesis and VQ-VAE-derived neural audio codecs, including universal compression with residual vector quantization. The proposed method builds on these approaches while introducing periodic activations, projected codebook learning, and a stabilized loss recipe.
- High-fidelity audio synthesis: GAN-based neural vocoders use feedforward generators for fast inference and multi-scale discrimination to penalize structure at different audio resolutions.MelGAN also introduced feature matching loss based on L1 distances between discriminator features.
- Neural audio compression models: VQ-VAEs became the dominant paradigm for neural audio codecs, with early speech codecs operating at 1.6 kbps.The early codec used a convolutional encoder and a WaveNet decoder.
- Neural audio compression models: SoundStream extended neural compression to diverse audio types and varying bitrates using one model with causal convolutions and residual vector quantization.Its training formulation combined adversarial, feature-matching, and multi-scale spectral reconstruction losses.
- Proposed method: The proposed method retains convolutional encoder-decoder, residual vector quantization, adversarial, and perceptual-loss components while adding three key design changes.These changes are periodic inductive bias through Snake activations, low-dimensional projection for codebook learning, and a stable loss recipe with fixed weights.
- Language modeling of natural signals: Neural language models motivate discrete audio tokens, but self-attention has quadratic sequence-length cost that is unsuitable for very high-dimensional natural signals.The passage identifies neural compression as a key component for modeling such signals.
3 The Improved RVQGAN Model
Improved RVQGAN combines convolutional encoding, residual vector quantization, periodic activations, improved codebook learning, quantizer dropout, and frequency-domain adversarial and reconstruction losses. These choices target high-fidelity, broadly applicable compression while retaining compact latent sequences and variable bitrate operation.
- Model framework: Residual vector quantization produces discrete codes while quantizer dropout allows one model to operate at several target bitrates.The encoder-decoder temporally downsamples audio before recursively quantizing residuals with distinct codebooks.
- Compression representation: Higher compression and lower latent frame rates support shorter sequences for language-model training while preserving audio quality relative to baselines.The target bitrate is an upper bound because the models support variable bitrates.
- 3.1 Periodic activation function: Replacing Leaky ReLU with Snake activations significantly improves audio fidelity by adding a periodic inductive bias to the generator.Snake includes a periodic component whose frequency is controlled by α.
- 3.2 Improved residual vector quantization: Factorized and L2-normalized codes improve codebook usage, bitrate efficiency, and reconstruction quality without k-means initialization or randomized restarts.Factorized lookup uses a low-dimensional space while code embeddings remain high-dimensional.
- 3.3 Quantizer dropout rate: Applying quantizer dropout with probability p = 0.5 narrows the full-bandwidth quality gap while preserving lower-bitrate reconstruction quality.Standard dropout can degrade full-bandwidth reconstruction quality.
- Adversarial and reconstruction losses: A complex multi-band, multi-scale STFT discriminator improves phase modeling, high-frequency prediction, and mitigation of aliasing artifacts.The model also uses multi-period waveform discrimination, multi-scale mel reconstruction losses, HingeGAN loss, and L1 feature matching.
4 Experiments
Experiments evaluate the codec through objective metrics, listening tests, ablations, and comparisons with competing codecs. The results show that specific architectural, adversarial, reconstruction, quantization, and sampling choices improve fidelity, bitrate efficiency, or artifact suppression.
- Evaluation setup: 3000 ten-second segments balance speech, music, and environmental sounds across evaluation sources and held-out speakers.The test set contains 1000 segments from each domain.
- Model and training recipe: The codec uses a convolutional encoder, residual vector quantizer, and convolutional decoder with downsampling rates [2] [8].The final architecture has 76M parameters, including a 1536-dimensional decoder.
- Ablation study: 99% to 62% bitrate efficiency and 9.12 to 1.07 SI-SDR drops occur when adversarial losses are removed, while spectral metrics remain relatively unaffected.The reconstruction-only model also produces buzzing artifacts because it does not learn to reconstruct phase.
- Ablation study: 7.68 from 9.12 SI-SDR results when low-hop reconstruction is replaced by single-scale high-hop mel reconstruction.The low-hop loss is retained because it improves waveform loss and modeling of fast transients and high frequencies.
- Ablation study: 8 is the optimal codebook latent dimension, while values such as 2 and 256 significantly worsen metrics and bitrate efficiency.Lower bitrate efficiency effectively lowers bandwidth and harms generator modeling capability.
5 Conclusion
The paper presents a high-fidelity universal neural audio compression algorithm that maintains audio quality across diverse audio types while achieving high compression rates. It also identifies domain-specific reconstruction limits, especially for environmental sounds and some musical instruments.
- The proposed algorithm combines audio generation, vector quantization, adversarial, and reconstruction advances for high-fidelity universal compression.
- The model maintains audio quality across various audio domains while achieving remarkable compression rates.
- The authors release thorough ablations, open-source code, and trained model weights to support future high-fidelity audio modeling.
- The codec performs best on speech and has more reconstruction issues with environmental sounds.
- Some musical instruments, including glockenspiel and synthesizer sounds, are not modeled perfectly.
A Appendix
The appendix describes a modified codebook-learning quantization operation that projects encoder outputs into a lower-dimensional codebook space before reconstructing quantized representations. Its loss balances reconstruction and codebook-related terms using stop-gradient behavior.
- Modified codebook learning algorithm: The modified quantization operation uses projection matrices to map encoder outputs into an intermediate representation and then into quantized representations.W_in maps the encoder output to the intermediate representation, while W_out maps it to z_q(x).
- Modified codebook learning algorithm: The codebook dimension M is much smaller than the encoder output dimension D.
- Modified codebook learning algorithm: The vector quantizer loss measures reconstruction error through a combination of two terms.
- Modified codebook learning algorithm: The stop-gradient operator prevents gradients from back-propagating through the codebook embedding e_k.
- Modified codebook learning algorithm: The hyperparameter β controls the balance between the two terms in the vector quantizer loss.