Source-linked AI summary
ClariNet: Parallel Wave Generation in End-to-End Text-to-Speech
Wei Ping, Kainan Peng, Jitong Chen
TL;DR
WaveNet provides high-fidelity waveform synthesis but is slow autoregressively, while prior parallel distillation can be unstable for highly peaked distributions. The paper distills a Gaussian IAF with closed-form regularized KL divergence and introduces a fully convolutional text-to-wave TTS model, which outperforms the separately trained pipeline and supports a competitive parallel vocoder.
Problem
WaveNet’s autoregressive inference is prohibitively slow, and parallel-WaveNet distillation can be unstable for highly peaked output distributions.
Method
The paper models raw waveforms with a variance-bounded Gaussian WaveNet, distills a Gaussian IAF using regularized closed-form KL divergence, and trains a fully convolutional text-to-wave model end to end.
Results
The text-to-wave model significantly outperforms the separately trained pipeline in naturalness, while its parallel vocoder achieves competitive results with an autoregressive vocoder.
Takeaways & Limitations
Gaussian IAF distillation enables simplified, stabilized parallel waveform synthesis within an end-to-end text-to-wave TTS architecture.
Takeaways & Limitations
Direct maximum-likelihood learning of IAF remains very slow because recovering latent variables from observed audio is autoregressive.
Abstract
from arXiv · showhide
In this work, we propose a new solution for parallel wave generation by WaveNet. In contrast to parallel WaveNet (van den Oord et al., 2018), we distill a Gaussian inverse autoregressive flow from the autoregressive WaveNet by minimizing a regularized KL divergence between their highly-peaked output distributions. Our method computes the KL divergence in closed-form, which simplifies the training algorithm and provides very efficient distillation. In addition, we introduce the first text-to-wave neural architecture for speech synthesis, which is fully convolutional and enables fast end-to-end training from scratch. It significantly outperforms the previous pipeline that connects a text-to-spectrogram model to a separately trained WaveNet (Ping et al., 2018). We also successfully distill a parallel waveform synthesizer conditioned on the hidden representation in this end-to-end model.
1 Introduction
The paper addresses WaveNet’s slow autoregressive inference and unstable parallel-WaveNet distillation by introducing Gaussian flow-based generation and a fully convolutional text-to-wave architecture. The resulting systems simplify training, enable end-to-end training from scratch, and improve or match prior waveform-synthesis pipelines.
- Motivation: WaveNet’s autoregressive inference is prohibitively slow because each sample must be generated before the next time step.Real-time high-fidelity synthesis therefore requires highly engineered inference kernels.
- Motivation: Parallel WaveNet’s KL-divergence training can produce high gradient variance and unstable training for highly peaked output distributions.Its use of mixture-of-logistics and logistic IAF distributions contributes to this distillation challenge.
- Parallel wave generation: The proposed method uses a variance-bounded Gaussian WaveNet and distills a Gaussian IAF by minimizing regularized KL divergence in closed form.A single Gaussian models raw waveforms without degrading audio quality, while closed-form estimation simplifies distillation and stabilizes training.
- Text-to-wave architecture: The first fully convolutional text-to-wave TTS architecture trains end to end from scratch by conditioning WaveNet on hidden states rather than mel-spectrograms.The text-to-wave model significantly outperforms the separately trained text-to-spectrogram-plus-vocoder pipeline in naturalness.
- Text-to-wave architecture: The parallel vocoder conditioned on the end-to-end model’s learned hidden representation achieves competitive results with an autoregressive vocoder.This extends parallel waveform synthesis to the proposed text-to-wave architecture.
2 Related Work
Prior neural TTS systems commonly separated text-to-spectrogram prediction from waveform synthesis, while normalizing flows and knowledge distillation offered relevant foundations. This work applies Gaussian IAF distillation to make parallel waveform generation practical within an end-to-end TTS model.
- Neural TTS: Earlier neural TTS systems retained multi-stage pipelines or predicted spectrograms that required Griffin-Lim or a separately trained neural vocoder.The paper positions its text-to-wave architecture as the first neural TTS model that directly generates waveforms from text.
- Neural vocoders: WaveNet vocoders can condition on linguistic features, F0, phoneme durations, mel-spectrograms, or the hidden representation of an end-to-end model.The paper evaluates parallel waveform synthesis with both mel-spectrogram and hidden-representation conditioning.
- Normalizing flows: Normalizing flows transform a simple initial distribution into a more complex one through invertible transformations, while IAF uses autoregressive networks for those transformations.IAF can reuse autoregressive architectures such as WaveNet.
- Knowledge distillation: Direct maximum-likelihood learning of IAF can be very slow because computing the corresponding latent variable from observed audio is autoregressive.The paper instead distills a Gaussian IAF from a pretrained autoregressive model using a numerically stable KL-divergence variant.
- Knowledge distillation: Knowledge distillation trains a smaller student network by minimizing discrepancies between teacher and student outputs.Parallel WaveNet applies this framework to distill a non-autoregressive student from an autoregressive WaveNet.
3 Parallel Wave Generation
The section presents Gaussian WaveNet and Gaussian IAF models for parallel waveform generation, using flow transformations and distillation to replace slow autoregressive sampling. The method regularizes KL matching and computes the Gaussian KL divergence in closed form.
- Gaussian Autoregressive WaveNet: WaveNet factorizes the waveform distribution into conditional distributions, taking previous samples and a conditioner as input to predict each current sample.The conditioner can be mel-spectrograms or hidden states.
- Gaussian Autoregressive WaveNet: A single Gaussian output distribution is used for the autoregressive WaveNet, trained by maximum likelihood estimation without degrading audio quality.The model predicts mean and standard deviation, operating on log standard deviation for numerical stability.
- Gaussian Inverse Autoregressive Flow: Gaussian IAF transforms white-noise latents into waveform samples in parallel, while its autoregressive network defines the shifting and scaling functions.The resulting waveform variables remain jointly modeled through latent-variable marginalization.
- Regularized KL Divergence: Gaussian distillation requires one latent sample and computes the per-time-step KL divergence in closed form, unlike logistic-to-mixture-of-logistics distillation, which requires sampling to estimate an intractable integral.Algorithm 1 produces both waveform samples and the student Gaussian parameters used for teacher comparison.
- Regularized KL Divergence: The student IAF is distilled from the teacher WaveNet by matching their output distributions with regularized KL divergence, which stabilizes optimization for highly peaked distributions.The regularization rapidly matches student and teacher standard deviations while preserving equality only when the distributions match.
- STFT Loss: KL loss alone can produce whisper voices, so the method adds a frame-level loss comparing student output samples with corresponding ground-truth audio.The authors contrast this with average power loss, whose effectiveness is tied to short training clips.
4 Text-to-Wave Architecture
The paper introduces a fully convolutional text-to-wave architecture that directly passes learned hidden representations through a bridge-net to a waveform vocoder, enabling end-to-end training from scratch. Its components include an encoder, decoder, bridge-net, and vocoder, with convolutional processing throughout.
- Architecture: The architecture directly feeds hidden representations from a text-to-spectrogram model through intermediate processing to WaveNet, avoiding mel-spectrogram conditioning for end-to-end training.The authors identify hidden-representation conditioning as crucial for successful training from scratch.
- Architecture: The encoder converts textual features into an internal representation, while the causal decoder autoregressively predicts log-mel spectrograms using attention.
- Bridge-net: The non-causal bridge-net processes decoder states, uses future context, and upsamples representations from frame-level to sample-level before waveform synthesis.It maps frame-level hidden representations to sample-level through convolution and transposed-convolution blocks.
- Vocoder: The vocoder is a Gaussian autoregressive WaveNet conditioned on the bridge-net output, with the option to replace it by a distilled student IAF.
- Training: All modules are convolutional, and the objective combines decoder, bridge-net, and vocoder losses with equal coefficients in the experiments.The basic convolution block uses a gated linear unit and a residual connection.
5 Experiment
Experiments evaluate Gaussian WaveNet, distilled Gaussian IAF students, and the proposed text-to-wave model on speech synthesis. The parallel student runs about 20 times faster than real time, while the text-to-wave model is compared with a separately trained pipeline.
- Data: The experiments use about 20 hours of English speech from one female speaker, downsampled from 48 kHz to 24 kHz.
- Autoregressive WaveNet: Gaussian autoregressive WaveNet achieves naturalness comparable to MoG and softmax outputs and outperforms MoL in the reported experiments.The study also reports conditional log-likelihoods on 16 test audios for continuous-output WaveNets.
- Student Gaussian IAF: The authors distill two 60-layer parallel student networks from a pretrained 20-layer Gaussian autoregressive WaveNet.Student-2 reverses the generated sequence between successive IAF blocks and reports improved performance.
- Student Gaussian IAF: ∼20 times faster than real time is the reported inference speed of the parallel student-net on an NVIDIA GeForce GTX 1080 Ti.
- Text-to-Wave Model: The text-to-wave model is trained from scratch and compared with the separately trained Deep Voice 3 pipeline using the same text preprocessing and single-speaker encoder-decoder hyperparameters.The text-to-wave system uses a six-layer bridge-net with input and output size 256.
6 Conclusion
The paper presents Gaussian IAF distillation and a fully convolutional text-to-wave TTS architecture as complementary approaches to parallel waveform generation. It reports stabilized distillation, end-to-end training from scratch, improved text-to-wave performance over a separate pipeline, and competitive parallel-vocoder results.
- Conclusion: A single variance-bounded Gaussian is sufficient for modeling raw WaveNet waveforms without degradation of audio quality.
- Conclusion: The Gaussian IAF student is distilled using a regularized KL divergence for peaked distributions, computed in closed form to simplify and stabilize training.
- Conclusion: The fully convolutional text-to-wave architecture supports end-to-end TTS training from scratch and outperforms the separately trained pipeline.
- Conclusion: A parallel neural vocoder conditioned on the end-to-end model’s hidden representation obtains competitive results with an autoregressive vocoder.
A Clipping Log-scale at Training
Training Gaussian WaveNet requires clipping the log-scale parameter to avoid numerical instability. The clipping constant affects likelihood, convergence, and speech quality, with diminishing likelihood gains beyond −9 in the reported study.
- Training stability: Clipping log σ is important because removing the clipping trick makes Gaussian WaveNet optimization numerically unstable.The clipping is applied during training but not inference.
A.1 Autoregressive WaveNet
Gaussian WaveNet’s clipping constant trades off likelihood, convergence speed, and numerical stability. A clipping constant of −9 is suggested as the default across tested datasets and conditioners.
- A.1 Autoregressive WaveNet: Smaller log σ clipping constants generally increase likelihood but require more iterations to converge.Validation likelihood improves substantially from −7 to −9, but only negligibly from −9 to −10.
- A.1 Autoregressive WaveNet: Removing clipping causes numerical instability, while uniform-noise dequantization behaves similarly to clipping at −10.
- A.1 Autoregressive WaveNet: The authors suggest −9 as the default Gaussian-teacher clipping constant across English and Mandarin datasets and several conditioner types.
- A.1 Autoregressive WaveNet: The predicted log σ distributions are compared before clipping for different clipping constants during training.
A.2 Gaussian IAF
Gaussian IAF distillation uses clipped log σ values for numerical stability while retaining an unclipped regularization term. The implementation combines dilated convolution blocks with practical stabilization heuristics.
- A.2 Gaussian IAF: During KL computation, log σp and log σq are clipped for numerical reasons, but clipping is excluded from the regularization term.
- A.2 Gaussian IAF: Larger KL clipping constants stabilize optimization but can make the KL-divergence loss less useful; −6 is suggested as the default.
- A.2 Gaussian IAF: Distillation stability can be improved by masking initially large KL losses, clipping gradients, increasing batch size, and lowering the learning rate.
- A.2 Gaussian IAF: Each dilated convolution stack uses ten layers with doubling dilation from 1 through 512.
- A.2 Gaussian IAF: The dilated convolution block computes hidden layers with gated convolutions using sigmoid and tanh branches, causal dilated convolutions, and an upsampled conditioner.
C Estimate the Sequence-level KL Divergence
The sequence-level KL divergence is expressed as an expectation of per-time-step Gaussian KL terms under samples generated by the student’s IAF. Summing those terms gives an unbiased estimate of the sequence-level divergence.
- C Estimate the Sequence-level KL Divergence: For an IAF student, sequence-level KL divergence is evaluated by sampling z from N(0,I), generating x through x=f(z), and summing conditional per-time-step KL terms.
- C Estimate the Sequence-level KL Divergence: The summed per-time-step KL divergence is an unbiased estimate of the sequence-level KL divergence.
- C Estimate the Sequence-level KL Divergence: The per-time-step terms use the closed-form KL divergence between Gaussian teacher and student distributions.