Source-linked AI summary
BigCodec: Pushing the Limits of Low-Bitrate Neural Speech Codec
Detai Xin, Xu Tan, Shinnosuke Takamichi, Hiroshi Saruwatari
TL;DR
Neural speech codecs experience significant reconstruction degradation around 1 kbps, while model capacity has not been explored as a source of further improvement. BigCodec scales the codec to 159M parameters and uses a single codebook with 8192 codes for 1.04 kbps speech coding. BigCodec at 1.04 kbps significantly outperforms previous low-bitrate codecs, matches codecs at 4–6 times higher bitrates objectively, and achieves higher subjective perceptual quality than ground truth.
Problem
Neural speech codecs experience significant reconstruction degradation around 1 kbps, while model capacity has not been explored as a source of further improvement.
Source: §I · p. 1Method
BigCodec scales the codec to 159M parameters and uses a single codebook with 8192 codes for 1.04 kbps speech coding.
Source: §I · p. 1Results
BigCodec at 1.04 kbps significantly outperforms previous low-bitrate codecs, matches codecs at 4–6 times higher bitrates objectively, and achieves higher subjective perceptual quality than ground truth.
Source: §I · p. 1, B. Main results · p. 4, §V · p. 4Takeaways & Limitations
Abstract
from arXiv · showhide
We present BigCodec, a low-bitrate neural speech codec. While recent neural speech codecs have shown impressive progress, their performance significantly deteriorates at low bitrates (around 1 kbps). Although a low bitrate inherently restricts performance, other factors, such as model capacity, also hinder further improvements. To address this problem, we scale up the model size to 159M parameters that is more than 10 times larger than popular codecs with about 10M parameters. Besides, we integrate sequential models into traditional convolutional architectures to better capture temporal dependency and adopt low-dimensional vector quantization to ensure a high code utilization. Comprehensive objective and subjective evaluations show that BigCodec, with a bitrate of 1.04 kbps, significantly outperforms several existing low-bitrate codecs. Furthermore, BigCodec achieves objective performance comparable to popular codecs operating at 4-6 times higher bitrates, and even delivers better subjective perceptual quality than the ground truth.
I. INTRODUCTION
BigCodec targets severe quality degradation near 1 kbps by scaling model capacity and simplifying quantization, achieving strong low-bitrate and higher-bitrate comparisons.
I. INTRODUCTION
- Current neural codecs commonly operate at 6–12 kbps, while information-rate estimates suggest speech may support substantially lower bitrates.The cited estimates place lexical information near 50 bps and an upper bound near 100 bps.Source: §I · p. 1
- BigCodec addresses low-bitrate degradation by scaling model size, a factor not explored by prior approaches focused on architectures, coding methods, or nondeterministic encoding.The motivation draws on scaling successes in language generation, image synthesis, and audio generation.Source: §I · p. 1
- BigCodec uses 159M parameters and a single 8192-codebook VQ-VAE at 1.04 kbps, avoiding the extra complexity of multiple codebooks.It is trained on 960 hours of LibriSpeech data.Source: §I · p. 1
- 1.04 kbps BigCodec significantly outperforms previous low-bitrate codecs and matches the objective performance of codecs operating at 4–6 times higher bitrates.Its evaluations include objective and subjective tests, ablations, unseen-language generalization, code utilization, and inference efficiency.Source: §I · p. 1
- BigCodec reports subjective perceptual quality exceeding ground-truth samples.This claim is part of the paper’s comprehensive evaluation results.Source: §I · p. 1
II. RELATED WORK
Prior low-bitrate codecs improve quality through stronger decoders, temporal-dependency modeling, predictive coding, and domain-specific optimization.
II. RELATED WORK
- Low-bitrate codecs have enhanced decoding with powerful audio-generation models and reduced transmitted information through nondeterministic encoding.These approaches target codecs operating around 1 kbps and often restrict training data to specific domains.Source: §II · p. 1
- Temporal modeling has been addressed with pretrained Transformers and predictive coding that autoregressively encodes temporal residuals.TF-Codec reduces redundancy over time compared with independent frame processing.Source: §II · p. 2
III. BIGCODEC
BigCodec combines a GAN-based VQ-VAE with convolutional and recurrent temporal modeling, single-code quantization, and low-dimensional projections to improve codebook utilization.
III. BIGCODEC
- BigCodec adopts a GAN framework containing a VQ-VAE generator and multiple discriminators, with architecture, training objective, and scaling described in sequence.Source: §III · p. 2
1) VQ-VAE generator:
- The encoder uses residual CNN blocks with snake activations and a two-layer unidirectional LSTM, while the mirrored decoder reconstructs waveforms through transposed convolutions.The LSTM models long-distance dependency, and the CNN blocks downsample or upsample the waveform.Source: 1) VQ-VAE generator · p. 2
- The encoder maps waveform x to hidden variables h, and the decoder maps quantized representations z_d to reconstructed waveform x̂ after an overall downsampling rate R.The encoder and decoder use matching downsampling and upsampling rates.Source: 1) VQ-VAE generator · p. 2
- Single-codebook quantization limits bitrate and complexity, while low-dimensional projection and L2 normalization improve codebook utilization through cosine-distance matching.The projected quantized vectors are mapped back to h’s original dimension before decoding.Source: 1) VQ-VAE generator · p. 2
2) Discriminators:
BigCodec uses multi-period and multi-scale STFT discriminators to model complementary temporal and spectral speech structure.
2) Discriminators:
- BigCodec adopts MPD for multiple periodic waveform patterns and MS-STFT for spectral structures at multiple resolutions.MSD was not used because preliminary experiments found no additional benefit.Source: 2) Discriminators · p. 2
B. Training objective
BigCodec combines perceptually motivated reconstruction, GAN, and vector-quantization losses, then scales its convolutional architecture from a compact baseline to a 159M-parameter model.
B. Training objective
- The vector-quantization loss learns the codebook by gradient descent in a low-dimensional space and uses commitment regularization plus a straight-through estimator.Moving-average codebook updates are omitted, while the commitment loss limits excessive encoder output growth.Source: B. Training objective · p. 2
- BigCodec’s training objective weights multi-scale mel-spectrogram reconstruction most heavily, alongside GAN, feature-matching, vector-quantization, and commitment losses.The mel-spectrogram loss has weight 15, the commitment loss has weight 0.25, and all other losses have weight 1.Source: B. Training objective · p. 2
C. Scaling up the model size
- BigCodec-base starts with about 17M parameters, four convolutional blocks, and a 512-dimensional latent representation.The decoder mirrors the encoder architecture.Source: C. Scaling up the model size · p. 2
- Scaling to five convolutional blocks and larger channel widths produces a final BigCodec VQ-VAE with 159M parameters and 22M discriminator parameters.The downsampling rate remains fixed during scaling.Source: C. Scaling up the model size · p. 2
IV. EXPERIMENTS
The experiments evaluate BigCodec on LibriSpeech using objective and subjective measures, with comparisons against popular and low-bitrate codecs under specified bitrate and training-data conditions.
A. Setup
- The primary evaluation uses LibriSpeech test-clean with 2,620 utterances at 16 kHz, after training on its 960-hour full training set.All models are trained for about 600k steps using randomly cropped one-second segments.Source: A. Setup · p. 3
- The theoretical BigCodec bitrate is 1.04 kbps, determined by a downsampling rate of 200 and an 8,192-entry codebook.The VQ projection dimension is set to 8.Source: A. Setup · p. 3
- BigCodec is compared with EnCodec at 1.5k and 6k bitrates, DAC at 1k and 4k, TF-Codec, and LLM-Codec.The comparison codecs differ in parameter counts, training corpora, and checkpoint sources.Source: A. Setup · p. 3
- Comparisons with SRCodec, SuperCodec, and LMCodec are omitted because of weak performance or too few available test samples.The omission is part of the reported experimental setup.Source: A. Setup · p. 3
- Objective evaluation covers bitrate utilization, spectral distortion, perceptual quality, intelligibility, and speaker similarity.SIM is computed from cosine similarity between ground-truth and reconstructed speaker embeddings.Source: A. Setup · p. 3
- Subjective quality is measured with MUSHRA using a hidden reference across 10 utterances, 30 recruited participants, and 29 valid participants.Each model receives 290 scores after excluding participants whose reference mean is below 80.Source: A. Setup · p. 3
B. Main results
On LibriSpeech, BigCodec outperforms previous low-bitrate codecs across objective metrics, matches higher-bitrate codec performance, and achieves a MUSHRA score slightly above ground truth.
B. Main results
- BigCodec consistently outperforms previous low-bitrate codecs across perceptual quality, intelligibility, and speaker similarity metrics.The evaluation uses the 2,620-utterance LibriSpeech test-clean set.Source: B. Main results · p. 3
- BigCodec achieves objective performance comparable to DAC-4k and EnCodec-6k despite operating at a much lower bitrate.The result is presented as evidence supporting the effectiveness of scaling up model size.Source: B. Main results · p. 4
- BigCodec’s MUSHRA score is 92.33, exceeding the ground-truth score of 91.84 and all baselines by a large margin.The subjective test reports an ICC(2) of 0.73, indicating good inter-rater agreement.Source: B. Main results · p. 4
- BigCodec’s approximated bitrate is 1.03 kbps versus a theoretical bitrate of 1.04 kbps, indicating high codebook utilization.The near match between the two bitrate values is attributed to the low-dimensional vector quantization design.Source: B. Main results · p. 4
C. Unseen languages
On seven unseen non-English languages, BigCodec remains stronger than previous low-bitrate codecs despite being trained only on monolingual English data.
C. Unseen languages
- BigCodec significantly outperforms all previous low-bitrate codecs on the seven-language MLS test set, despite some decline from in-domain performance.The test set contains 700 utterances from German, Dutch, Spanish, French, Italian, Portuguese, and Polish.Source: C. Unseen languages · p. 4
- BigCodec is the only evaluated codec trained on a monolingual corpus, making its multilingual result an out-of-domain generalization test.The training corpus is English while the evaluation languages exclude English.Source: C. Unseen languages · p. 4
- Speaker-similarity scores increase for almost all codecs on this test set, which the authors attribute to its relatively small number of speakers.This observation qualifies interpretation of SIM on the multilingual evaluation.Source: C. Unseen languages · p. 4
D. Ablation studies
The ablations show that BigCodec’s gains depend primarily on sufficient model capacity and temporal-dependency modeling, while further scaling model or data size provides no clear benefit.
D. Ablation studies
- BigCodec-base performs significantly worse than BigCodec, showing that scaling up model size is effective for low-bitrate speech coding.Despite its smaller size, BigCodec-base remains comparable to the best baseline, TF-Codec, which uses sophisticated autoregressive predictive coding.Source: D. Ablation studies · p. 4
- BigCodec-300M and BigCodec-ll60k achieve only comparable performance to BigCodec, indicating no further gains from increasing model or training-data size.The data-scaling result suggests no clear codec advantage from expanding beyond the LibriSpeech training setup.Source: D. Ablation studies · p. 4
- The small-enc variant performs slightly worse than BigCodec, indicating that a sufficiently large encoder is also necessary for optimal performance.Source: D. Ablation studies · p. 4
- Removing the LSTM causes underperformance, underscoring the importance of modeling temporal dependency in neural speech codecs.The comparison uses a 142M-parameter model after increasing the encoder’s final CNN channel size to avoid the unfairness of a nearly halved parameter count.Source: D. Ablation studies · p. 4
E. Efficiency analysis
BigCodec operates in real time on the tested CPU, but its higher performance comes with a substantial efficiency trade-off compared with the smaller BigCodec-base model.
E. Efficiency analysis
- 1.1× RTF for both encoding and decoding lets BigCodec process speech in real time on the tested Intel CPU.An RTF greater than one indicates real-time processing.Source: E. Efficiency analysis · p. 4
- BigCodec-base reaches 3.1× RTF, showing the efficiency cost associated with BigCodec’s higher-performance, more complex model.The reported RTFs were approximated using the entire 2620-utterance test set.Source: E. Efficiency analysis · p. 4
V. CONCLUSIONS
BigCodec advances low-bitrate speech coding through larger capacity and improved component design, achieving strong quality at 1.04 kbps across in-domain and unseen-language evaluations.
V. CONCLUSIONS
- At 1.04 kbps and 159M parameters, BigCodec significantly outperforms previous low-bitrate codecs on in-domain and unseen-language evaluations.Source: §V · p. 4
- BigCodec achieves objective performance comparable to codecs operating at 4-6 times higher bitrates and better subjective perceptual quality than ground-truth samples.Source: §V · p. 4
- The codec’s approach combines model scaling with optimized vector quantization and temporal-dependency modeling to improve low-bitrate speech coding.Source: §V · p. 4
- Future work will generalize BigCodec to arbitrary audio data and explore further bitrate reductions.Source: §V · p. 4