Source-linked AI summary
Wavenet based low rate speech coding
W. Bastiaan Kleijn, Felicia S. C. Lim, Alejandro Luebs, Jan Skoglund, Florian Stimberg, Quan Wang, Thomas C. Walters
TL;DR
Traditional parametric speech coding achieves low rates but can reproduce speech poorly because its signal model is limited. The paper uses WaveNet to generate speech from a standard 2.4 kb/s parametric bitstream and compares this with waveform coding using the same generative model. The results show high-quality parametric coding, a substantial rate penalty for waveform approximation, and speaker recognition that remains usable for unseen speakers.
Problem
Traditional parametric coding provides low rates but poor reconstruction quality because its signal model is inadequate.
Method
The paper uses WaveNet as a generic generative model in parametric and waveform coders, encoding conditioning variables alone or together with the observed waveform.
Results
The 2.4 kb/s parametric coder achieves quality similar to waveform coders operating at much higher rates.
Takeaways & Limitations
WaveNet-based coding can provide high-quality speech at low rates, while waveform approximation incurs a large rate penalty.
Takeaways & Limitations
WaveNet training and operation are computationally expensive compared with conventional coders, except for the parametric WaveNet encoder.
Abstract
from arXiv · showhide
Traditional parametric coding of speech facilitates low rate but provides poor reconstruction quality because of the inadequacy of the model used. We describe how a WaveNet generative speech model can be used to generate high quality speech from the bit stream of a standard parametric coder operating at 2.4 kb/s. We compare this parametric coder with a waveform coder based on the same generative model and show that approximating the signal waveform incurs a large rate penalty. Our experiments confirm the high performance of the WaveNet based coder and show that the speech produced by the system is able to additionally perform implicit bandwidth extension and does not significantly impair recognition of the original speaker for the human listener, even when that speaker has not been used during the training of the generative model.
1. INTRODUCTION
The paper targets high-quality speech coding at rates around 2–3 kb/s by replacing limited signal models with WaveNet-based generation. It distinguishes parametric coding, which transmits conditioning variables, from waveform coding, which also approximates the observed waveform.
- Motivation: The proposed approach aims to provide good speech quality at rates around 2–3 kb/s, where conventional systems often operate above 16 kb/s.The paper presents this as having potential to improve the rate-quality trade-off.
- Motivation: Speech coders typically use rates roughly two orders of magnitude above the estimated speech information rate of less than 100 b/s.The gap reflects substantial redundancy in common coding algorithms.
- Coding approaches: Parametric coders generate speech from model parameters, but reconstruction quality is limited by the model’s efficacy.Waveform coding can exploit conditioning information but transmits an approximation of the original waveform.
- Approach: WaveNet is used as a generic generative model for speech coding in two architectures: a parametric coder and a waveform coder.The parametric coder encodes only conditioning variables, whereas the waveform coder also encodes the observed waveform.
- Approach: The WaveNet coder is designed for speakers not present in training and can be generalized to the multi-speaker case.Its parametric coder is not speaker dependent and can use a conventional low-complexity decoder.
2. ALGORITHM
The paper uses WaveNet as a generic speech generator for both low-rate parametric coding and waveform coding, while analyzing the information rate produced by generation. The parametric architecture decodes conventional conditioning variables into speech, whereas waveform coding additionally represents the observed signal and trades rate for reconstruction fidelity.
- 2.1. Parametric WaveNet Coder: The parametric coder transmits only conventional conditioning variables, which a WaveNet decoder uses to generate the speech signal.The conditioning variables include spectral envelope, pitch, and voicing level; the selected Codec 2 stream operates at 2.4 kb/s.
- 2.1. Parametric WaveNet Coder: The decoder uses 8 kHz conditioning variables with 16 kHz speech targets, enabling implicit bandwidth extension.This design addresses the narrow-band operation of most parametric coders while producing wide-band output.
- 2.1. Parametric WaveNet Coder: WaveNet models the conditional distribution of each next sample and samples it to generate the output waveform.The network is trained with the same conditioning variables used at run time and with many talkers without speaker-label conditioning.
- 2.2. Rate Analysis: The generated signal rate equals the conditioning-parameter rate plus the information rate introduced by the generative process.For WaveNet, the generated information rate is evaluated from the conditional entropy of its sampled output distribution under a stationarity and ergodicity assumption.
- 2.3. WaveNet Waveform Coder: Waveform coding conveys additional waveform information and is therefore inefficient relative to relying on a generative model when the model adequately describes the signal.The framework can evaluate input log likelihood to select between low-rate parametric and higher-rate waveform coding, although the generative coder cannot guarantee output quality in all situations.
- 2.3. WaveNet Waveform Coder: The WaveNet waveform coder is close to optimal for squared error on samples with µ-law-warped amplitude, with scalar quantization imposing an asymptotic penalty of at most 1.5 dB or 0.25 bits per sample.Removing this penalty would require higher-dimensional conditional distributions or analysis-by-synthesis, both of which introduce substantial complexity.
3. EXPERIMENTAL RESULTS
The experiments evaluate WaveNet coders at 2.4 kb/s using objective quality, subjective listening, information-rate, and speaker-identification tests. The parametric WaveNet coder achieves high subjective quality at low rate, while waveform generation requires substantially more rate and speaker recognition remains impaired but possible.
- Experimental setup: 2.4 kb/s Codec 2 conditioning variables drove WaveNet decoding from 16 kHz output, with training and testing performed on speakers with no overlap.The evaluation used 32,580 training utterances from 123 speakers and 2,907 test utterances from 8 speakers.
- Speech information rates: 42 kb/s was required for waveform generation, compared with the 2.4 kb/s conditioning rate, demonstrating a large rate penalty for transmitting waveform information.The estimated rates were 2.65 bits per sample, or 42 kb/s at 16 kHz; related estimates were 2.61 and 2.62 bits per sample.
- Speech information rates: The waveform coder’s instantaneous rate was higher for relatively unstructured fricatives and lower for voiced segments, with additional variation across pitch cycles.The highest rate was not associated with the pitch pulse, suggesting accurate pitch prediction but noisier waveform reconstruction during parts of the pitch cycle.
- Quality experiments: The parametric WaveNet coder joined the high-quality group in MUSHRA tests with AMR-WB and the WaveNet waveform coder, unlike Speex, Codec 2, and MELP.Twenty-one participants evaluated eight utterances, and the parametric coder had subjective quality similar to existing waveform coders at significantly lower rates.
- Speaker identification experiments: 15.8% verification EER was obtained for WaveNet-coded speech versus 8.4% for µ-law-coded speech in the objective speaker-identification test.The test used overlapping speakers but non-overlapping enrollment and verification utterances; the authors attribute the increase to restricted spectral resolution at low rate.
- Speaker identification experiments: 41% of triangle-test trials correctly identified the different speaker, compared with 33% expected if the speakers were indistinguishable.The test used 15 listeners and 16 trials, comparing models trained with and without overlapping speakers; the discrepancy was expected to diminish with more training speakers.
4. CONCLUSIONS
WaveNet’s high-fidelity conditional distribution enables a 2.4 kb/s parametric speech coder whose quality is comparable to much higher-rate waveform coders. The approach remains computationally costly, while further conditioning and filtering refinements may improve performance.
- 2.4 kb/s parametric WaveNet speech coding achieves quality similar to waveform coders operating at much higher rates.
- Waveform WaveNet coding can be combined with a likelihood-based quality measure to switch from parametric coding when appropriate.
- WaveNet training and execution have higher computational costs than conventional coders, although the parametric encoder remains low complexity.
- Further improvements may come from refining conditioning parameters and interpolation, adding pre- and postfiltering, and examining redundant long-lag memory.