Source-linked AI summary
Parallel WaveNet: Fast High-Fidelity Speech Synthesis
Aaron van den Oord, Yazhe Li, Igor Babuschkin, Karen Simonyan, Oriol Vinyals, Koray Kavukcuoglu, George van den Driessche, Edward Lockhart, Luis C. Cobo, Florian Stimberg, Norman Casagrande, Dominik Grewe, Seb Noury, Sander Dieleman, Erich Elsen, Nal Kalchbrenner, Heiga Zen, Alex Graves, Helen King, Tom Walters, Dan Belov, Demis Hassabis
TL;DR
WaveNet produces highly realistic speech but its sequential generation makes real-time deployment difficult. The paper introduces Probability Density Distillation to train a parallel feed-forward student from a trained WaveNet teacher. The resulting system maintains speech quality, achieves large speed-ups, supports multiple languages and speakers, and serves Google Assistant queries in production.
Problem
WaveNet's sequential sample generation makes its high-quality speech synthesis difficult to deploy in real time.
Method
Probability Density Distillation trains a parallel feed-forward WaveNet student using an already trained WaveNet as teacher.
Results
The distilled system achieves several orders of magnitude speed-up with no significant quality difference, and transfers to new languages and multiple speakers.
Takeaways & Limitations
The resulting system is deployed by Google to serve Google Assistant queries in real time to millions of users.
Takeaways & Limitations
Experiments use model-predicted pitch and linguistic features rather than ground-truth speech-derived pitch or duration.
Abstract
from arXiv · showhide
The recently-developed WaveNet architecture is the current state of the art in realistic speech synthesis, consistently rated as more natural sounding for many different languages than any previous system. However, because WaveNet relies on sequential generation of one audio sample at a time, it is poorly suited to today's massively parallel computers, and therefore hard to deploy in a real-time production setting. This paper introduces Probability Density Distillation, a new method for training a parallel feed-forward network from a trained WaveNet with no significant difference in quality. The resulting system is capable of generating high-fidelity speech samples at more than 20 times faster than real-time, and is deployed online by Google Assistant, including serving multiple English and Japanese voices.
1 Introduction
WaveNet delivers highly natural speech but generates audio sequentially, limiting real-world deployment. Probability Density Distillation trains a parallel feed-forward model that preserves quality while greatly increasing synthesis speed.
- WaveNet achieves state-of-the-art speech synthesis but has prohibitive generation speed for real-world deployment.
- Probability Density Distillation transfers knowledge from a trained WaveNet teacher to a feed-forward IAF student.
- Parallel WaveNet shows no loss in perceived quality compared with the original while remaining superior to previous benchmarks.
- More than 1000× speed-up is achieved for sample generation relative to the original WaveNet.
2 WaveNet
WaveNet models raw audio with causal convolutions that train in parallel but generate samples sequentially. Its dilated convolutions expand receptive fields, while higher-fidelity audio modelling uses 16-bit samples, mixture-of-logistics distributions, and 24kHz sampling.
- WaveNet models autoregressive audio distributions with causal convolutions, producing conditional outputs in one forward pass.
- At generation time, WaveNet must sample each waveform value before obtaining later inputs, making real-time synthesis challenging.
- Dilated causal convolutions let WaveNet expand its receptive field exponentially with network depth.
- WaveNet uses gated activations and conditioning on extra information such as linguistic features.
- The production model improves fidelity by modelling 16-bit audio with discretized mixtures of logistics and increasing sampling from 16kHz to 24kHz.
3 Parallel WaveNet
Parallel WaveNet uses inverse autoregressive flows to generate all elements of a high-dimensional audio sample in parallel, replacing WaveNet’s sequential sampling bottleneck. Stacking up to four flow networks improves quality while preserving parallel generation.
- Inverse autoregressive flows: Inverse autoregressive flows arrange latent variables so that all elements of a high-dimensional audio sample can be generated in parallel.They transform a tractable latent distribution through an invertible nonlinear mapping with a triangular Jacobian.
- Flow architecture: The parallel model starts from unconditional logistic noise and passes each network’s output to the next flow.The first network receives x0 = z, and subsequent networks iteratively transform the previous output.
- Flow architecture: The network outputs a sample x together with location and scale parameters, defining each conditional output as a logistic distribution.The conditional distribution p(xt|z<t) is parameterised by µt and st.
- Flow architecture: The flow uses the same convolutional autoregressive network structure as WaveNet, while changing the inputs and outputs for inverse autoregressive generation.With the same output distribution class, the two architectures can mathematically model the same multivariate distributions.
- Flow architecture: Up to 4 flow iterations improved quality, and the final parallel WaveNet used separate weights for each flow.The flows were implemented by stacking networks on top of one another rather than sharing their weights.
4 Probability Density Distillation
Probability Density Distillation trains a parallel student from a trained WaveNet teacher because direct maximum-likelihood training would require slow sequential likelihood estimation. The objective matches teacher probabilities while preserving student entropy, supplemented by losses that guide speech power and perceptual quality.
- Motivation: Training the parallel WaveNet directly with maximum likelihood is impractical because likelihood estimation requires a sequential, slow inference procedure.The teacher-student setup transfers knowledge from an already trained WaveNet to a parallel student.
- Distillation objective: Probability Density Distillation trains a feed-forward student to match the probability of its samples under a trained WaveNet teacher.The teacher is held fixed, and the student cooperatively matches the teacher’s probabilities rather than fooling it adversarially.
- Distillation objective: The KL-divergence objective combines student-teacher cross-entropy with student entropy, whose inclusion prevents collapse to the teacher’s mode.When the KL-divergence reaches zero, the student distribution has recovered the teacher’s distribution.
- Distillation objective: The distillation estimator samples from the student while evaluating the teacher’s conditional probabilities in parallel, reducing variance relative to naive sample evaluation.The teacher’s mixture-of-logistics output makes the loss differentiable with respect to both the generated sample and its preceding samples.
- Additional losses: Additional losses guide the student toward speech with human-like frequency-band power and perceptually relevant pronunciation features.The power loss compares average short-time Fourier-transform power, while a classifier-based perceptual loss penalises bad pronunciations; style loss performed better than feature reconstruction in these experiments.
- Additional losses: Contrastive distillation matches teacher and student under the same conditioning while separating evaluations under different conditioning pairs.The contrastive-term weight was set to γ = 0.3 in the experiments.
5 Experiments
Experiments show that distilled parallel WaveNet preserves WaveNet’s speech quality while enabling substantially faster sampling, including multi-speaker and multilingual generation. Ablations identify complementary roles for the distillation losses and multiple flows.
- Experimental setup: The experiments used linguistically conditioned text-to-speech models with predicted pitch and did not use ground-truth pitch or duration from human speech.Test sentences were not present or similar to those used for training.
- Sampling speed: 500,000+ timesteps/second for distilled WaveNet contrasts with 172 timesteps/second for the autoregressive implementation.Both models were benchmarked on an NVIDIA P100 GPU with TensorFlow and XLA.
- Speech quality: 4.41 ± 0.08 MOS for distilled WaveNet matches the autoregressive teacher’s 4.41 ± 0.07 and exceeds the concatenative baseline’s 4.19 ± 0.1.The comparison used speech from a professional female speaker.
- Multi-speaker generation: The multi-speaker distilled model significantly outperforms both parametric and concatenative baselines for all evaluated English and Japanese speakers.The evaluation includes multiple English speakers and one Japanese speaker.
- Ablation studies: KL loss combined with power-loss produces natural speech, perceptual loss adds a small noticeable improvement, and contrastive loss reduces noise without further preference gains.The contrastive loss is important for production-quality speech because most raters do not notice the reduced noise.
- Ablation studies: 4.41 MOS with multiple flows exceeds the 4.21 MOS obtained with a single flow.The parallel WaveNet architecture uses multiple inverse-autoregressive flows.
6 Conclusion
The paper introduces Probability Density Distillation for high-fidelity speech synthesis and reports orders-of-magnitude speed-ups over original WaveNet without significant quality loss. The method also transfers to new languages and multiple speakers and has been deployed for real-time Google Assistant queries.
- Probability Density Distillation is introduced as a method for high-fidelity speech synthesis based on WaveNet.
- Orders-of-magnitude speed-up over original WaveNet is achieved with no significant difference in quality.
- The algorithm transfers successfully to new languages and multiple speakers.
- The resulting system is deployed in production for real-time Google Assistant queries serving millions of users.
A.1 Argument against MAP estimation
The paper argues against MAP estimation for distillation because selecting the teacher’s most likely samples can produce silence instead of perceptually realistic audio. This failure reflects the stochastic structure of the modeled data distribution, not adversarial teacher behavior.
- MAP estimation uses only cross-entropy, making generator samples as likely as possible under the teacher.
- Audio produced by MAP estimation can be almost completely silent, even when conditioned on linguistic features.
- The silence is a fundamental property of the approximated data distribution rather than adversarial behavior by the teacher.
- For white noise, MAP selects the mode 0 at every timestep, yielding silence instead of the distribution’s characteristic continual hiss.
A.2 Autoregressive Models and Inverse-autoregressive Flows
Autoregressive models and IAFs can represent the same distributions but impose different inductive biases and receptive-field requirements. The Fibonacci series illustrates that autoregressive modeling can capture certain dependencies more locally than an IAF.
- Autoregressive models and IAFs can in principle model the same distributions despite having different inductive biases.
- An autoregressive model represents the Fibonacci series with a receptive field of two using the preceding two values.
- An IAF requires a receptive field at least as large as the sequence position k to model the Fibonacci series correctly.