Source-linked AI summary
Low Bit-Rate Speech Coding with VQ-VAE and a WaveNet Decoder
Cristina Gârbacea, Aäron van den Oord, Yazhe Li, Felicia S C Lim, Alejandro Luebs, Oriol Vinyals, Thomas C Walters
TL;DR
Speech codecs need compact representations, yet high-quality codecs typically operate at bit rates far above speech's estimated information rate. This paper adapts VQ-VAE with a WaveNet decoder for low-bit-rate coding while targeting speaker and prosody preservation. At 1,600 bps, LibriSpeech perceptual quality is roughly halfway between MELP and AMR-WB; with a seen test speaker and studio-quality speech, quality is similar to AMR-WB.
Problem
High-quality speech codecs typically exceed 16 kbps despite speech's true information rate being estimated below 100 bps, and a generic codec must preserve speaker identity and prosody.
Source: §I · p. 1, §II · p. 2Method
The paper adapts VQ-VAE with a WaveNet decoder as an end-to-end codec, adding an utterance-level speaker code and auxiliary f0 prediction to encourage speaker and prosody preservation.
Source: §I · p. 1, §II-A · p. 2Results
At 1,600 bps on LibriSpeech, perceptual quality is roughly halfway between MELP at 2,400 bps and AMR-WB at 23.05 kbps; studio-trained output with a seen test speaker is similar to AMR-WB.
Source: §III-A · p. 3Takeaways & Limitations
The results support VQ-VAE's suitability as a very-low-bit-rate codec that can outperform some codecs at 2,400 bps, though speaker-identity preservation remains variable.
Source: §IV · p. 4The studio-quality result comparable to AMR-WB used a test speaker present in training, with both training and test audio of very high quality.
Source: §III-A · p. 3
Abstract
from arXiv · showhide
In order to efficiently transmit and store speech signals, speech codecs create a minimally redundant representation of the input signal which is then decoded at the receiver with the best possible perceptual quality. In this work we demonstrate that a neural network architecture based on VQ-VAE with a WaveNet decoder can be used to perform very low bit-rate speech coding with high reconstruction quality. A prosody-transparent and speaker-independent model trained on the LibriSpeech corpus coding audio at 1.6 kbps exhibits perceptual quality which is around halfway between the MELP codec at 2.4 kbps and AMR-WB codec at 23.05 kbps. In addition, when training on high-quality recorded speech with the test speaker included in the training set, a model coding speech at 1.6 kbps produces output of similar perceptual quality to that generated by AMR-WB at 23.05 kbps.
I. INTRODUCTION AND RELATED WORK
The paper frames speech coding as a high-rate problem with room for learned compression, and evaluates VQ-VAE with a WaveNet decoder as a low-bit-rate codec. Its modifications target prior VQ-VAE speech reconstructions that could preserve syllabic content while changing prosody.
I. INTRODUCTION AND RELATED WORK
- Deep-learning autoencoders learn encoder-decoder mappings from data; prior speech proposals included rates as low as 369 bps and a codec comparable to AMR-WB at 9–24 kbps.High-quality speech codecs typically operate above 16 kbps.Source: §I · p. 1
- Speech's true information rate is estimated below 100 bps, while current systems use roughly two orders of magnitude more for good quality.The authors describe this gap as suggesting room for improvement in speech coding.Source: §I · p. 1
- Prior work paired a Codec2 bitstream at 2.4 kbps with a learned WaveNet decoder for AMR-WB-like quality, while VQ-VAE provided a compact discrete latent representation for speech.VQ-VAE combines a VAE with vector quantization; categorical latent samples index embeddings passed to the decoder.Source: §I · p. 1
- The paper evaluates VQ-VAE as an end-to-end speech codec and modifies its architecture to target low bit rates while preserving content and prosody.It aims for a speaker-independent model with lower bit rates and higher reconstruction quality than then-current state of the art.Source: §I · p. 1
- Earlier VQ-VAE speech reconstructions used latent codes 64 times smaller than the waveform, but could alter prosody and were proposed for coding if generalized across speakers.The codes correlated with syllables, while the decoder generated plausible-sounding details.Source: §I · p. 2
II. VQ-VAE BASED SPEECH CODEC
The codec maps speech into a discrete latent space and reconstructs it with a decoder, with design changes intended to preserve speaker identity and prosody. A time-invariant code carries utterance-level information, while an auxiliary pitch-prediction task encourages prosodic information through the bottleneck.
II. VQ-VAE BASED SPEECH CODEC
- The codec must preserve both speaker identity and prosody while mapping audio to a discrete latent representation and reconstructing it with the decoder.The authors introduce architectural changes to meet these requirements.Source: §II · p. 2
II-A. Maintaining speaker identity
- Mean pooling encoder outputs across time creates a separate, time-invariant codebook intended to carry speaker information apart from time-varying message codes.The original explicit one-hot speaker conditioning is removed to encourage generalization to unseen speakers.Source: §II-A · p. 2
- The static code is computed over the entire utterance, adding 1–2 bytes but making that long window unrealistic for online coding.The authors suggest a slowly varying code based only on past information as an alternative.Source: §II-A · p. 2
II-B. Constraining prosody
- An auxiliary decoder predicts the training utterance's f0 track from the shared latent representation, encouraging pitch information to pass through the bottleneck to the waveform decoder.The f0 predictor can be removed at codec use, leaving the audio decoder to reconstruct the waveform.Source: §II-B · p. 2
- A stride-5 encoder convolution aligns 16 kHz audio with f0 features sampled at 200 Hz.Source: §II-B · p. 2
III. EXPERIMENTS
The experiments evaluate VQ-VAE speech coding on studio and LibriSpeech data using MUSHRA listening tests, then assess speaker similarity with MOS. At 1,600 bps, the LibriSpeech model scores roughly halfway between MELP and AMR-WB, while VQ-VAE speaker-similarity MOS exceeds two low-bitrate codecs.
III. EXPERIMENTS
- Experiments used LibriSpeech with disjoint train and test speakers, an augmented version adding about five hours of test-speaker audio, and a 300-hour Studio corpus.The augmented set held out 1/100 of test-clean for evaluation; Studio data were used to investigate performance in ideal training conditions.Source: §III · pp. 2–3
- MUSHRA tests assessed perceived codec quality using eight utterances rated by 100 human raters each, with codecs including MELP and AMR-WB as comparisons.The tests used a 0–100 scale and included a reference, hidden reference, and low-quality Speex anchor.Source: §III · p. 3
III-A. Quality Evaluation
- A 14-fold compression-rate improvement over a standard wideband codec brought minimal perceptual-quality reduction with studio training and a test speaker seen during training.The VQ-VAE, AMR-WB at 23.05 kbps, and 8-bit µ-law clustered at high reconstruction quality, while Speex, Codec2, and MELP at 2.4 kbps clustered lower.Source: Figure 1 · p. 3, §III-A · p. 3
- At 1,600 bps, LibriSpeech VQ-VAE scores roughly halfway between MELP at 2,400 bps and AMR-WB at 23.05 kbps; at 800 bps, it outperforms MELP.At 400 bps, performance falls below Speex at 2,400 bps.Source: §III-A · pp. 3–4, Figure 2 · p. 4
- Including test-speaker utterances in training produced a small perceptual-quality advantage for the 1.6 kbps VQ-VAE model.Figure 3 compares models trained on Studio, LibriSpeech, and LibriSpeech plus test-set speakers.Source: Figure 3 · p. 4, §III-A · p. 4
III-B. Speaker Transparency
- Speaker-similarity MOS was higher for VQ-VAE than for MELP and Speex, and slightly higher when test speakers were included in training.For one speaker, MOS rose from 1.85 without test-speaker training to 2.73 with it.Source: Table I · p. 4, §III-B · p. 4
IV. CONCLUSION
The adjusted VQ-VAE model shows initial suitability as a robust very-low-bitrate speech codec, outperforming some popular 2,400-bps codecs at 1,600 bps. Listening tests suggest better speaker-identity preservation than other low-rate codecs, but its variability remains unresolved.
IV. CONCLUSION
- At 1,600 bps, VQ-VAE outperforms some popular low-bitrate speech codecs operating at 2,400 bps.The authors describe these as initial results demonstrating the model’s suitability as a robust very-low-bitrate codec.Source: §IV · p. 4
- After architecture adjustments, the model is speaker- and prosody-transparent.The conclusion describes the adjusted architecture as making the model transparent to speaker and prosody.Source: §IV · p. 4
- Listening tests indicate somewhat better speaker-identity preservation than other low-rate codecs, but variability in preservation needs further study.Source: §IV · p. 4
- The authors expect model performance to improve beyond the current 1,600 bps setting.Source: §IV · p. 4