Source-linked AI summary
WaveGlow: A Flow-based Generative Network for Speech Synthesis
Ryan Prenger, Rafael Valle, Bryan Catanzaro
TL;DR
High-quality, real-time speech synthesis remains difficult because autoregressive vocoders are serial and computationally constrained. WaveGlow uses a flow-based network to synthesize speech from mel-spectrograms without autoregression, achieving over 500kHz on an NVIDIA V100 while matching the audio quality of a strong WaveNet implementation.
Problem
High-quality, real-time speech synthesis is challenging, and autoregressive vocoders are inherently serial, limiting efficient use of parallel processors.
Method
WaveGlow combines flow-based modeling with WaveNet-inspired architecture in a single network trained by maximizing likelihood.
Results
More than 500kHz synthesis speed and audio quality as good as the best publicly available WaveNet implementation were achieved on an NVIDIA V100.
Takeaways & Limitations
WaveGlow provides efficient speech synthesis with a simple model that is easy to train, supporting deployment of high-quality audio synthesis.
Takeaways & Limitations
The likelihood formulation assumes a spherical Gaussian variance, with the variance term contributing to the objective.
Abstract
from arXiv · showhide
In this paper we propose WaveGlow: a flow-based network capable of generating high quality speech from mel-spectrograms. WaveGlow combines insights from Glow and WaveNet in order to provide fast, efficient and high-quality audio synthesis, without the need for auto-regression. WaveGlow is implemented using only a single network, trained using only a single cost function: maximizing the likelihood of the training data, which makes the training procedure simple and stable. Our PyTorch implementation produces audio samples at a rate of more than 500 kHz on an NVIDIA V100 GPU. Mean Opinion Scores show that it delivers audio quality as good as the best publicly available WaveNet implementation. All code will be made publicly available online.
1. INTRODUCTION
WaveGlow addresses the challenge of synthesizing high-quality speech in real time by replacing serial autoregressive vocoding with a fast flow-based model. It uses one network and one likelihood objective while matching the audio quality of a strong WaveNet implementation.
- Motivation: High-quality real-time speech synthesis is difficult because audio has high dimensionality, long-term dependencies, and strict perceptual and computational constraints.Speech quality drops below 16 kHz, while higher sampling rates increase quality requirements.
- Limitations of prior approaches: Autoregressive vocoders model future samples from previous ones, making inference inherently serial and often slower than 16 kHz without quality loss.Their serial structure prevents full use of parallel processors such as GPUs and TPUs.
- Limitations of prior approaches: Non-autoregressive alternatives exceed 500 kHz on GPUs but are harder to train and implement because they use compound losses, multiple networks, or both.Parallel WaveNet and Clarinet require student and teacher networks, while the cited approaches also address mode collapse or audio quality with additional losses.
- WaveGlow: WaveGlow is a flow-based network that generates high-quality speech from mel-spectrograms without autoregression.It combines ideas from Glow and WaveNet.
- Results: More than 500 kHz on an NVIDIA V100 GPU and over 25 times real time are reported for WaveGlow, with audio quality matching the best publicly available WaveNet implementation on the same dataset.The quality comparison is based on Mean Opinion Scores.
2. WAVEGLOW
WaveGlow models audio conditioned on mel-spectrograms by transforming Gaussian noise through an invertible sequence of flow layers. Its coupling layers, invertible convolutions, likelihood objective, and multi-scale outputs provide a tractable and efficient generative architecture.
- Generative formulation: WaveGlow generates audio by transforming Gaussian samples into the distribution of audio samples conditioned on a mel-spectrogram.The Gaussian has zero mean and the same number of dimensions as the desired output.
- Likelihood training: Flow-based networks make likelihood training tractable by constraining each layer to be invertible and applying a change-of-variables calculation.The model is trained by minimizing the negative log-likelihood of the data.
- Likelihood training: The likelihood combines a spherical-Gaussian log-likelihood with log-determinant Jacobian terms from the transformations.The Jacobian term accounts for volume changes and prevents layers from optimizing the Gaussian norm by collapsing inputs.
- Architecture: WaveGlow groups eight audio samples into vectors, then applies flow steps consisting of an invertible 1x1 convolution followed by an affine coupling layer.The architecture follows Glow-style processing while incorporating WaveNet-inspired components.
- Architecture: Affine coupling layers leave one channel half unchanged while using it to produce scale and translation terms for the other half, preserving invertibility even when WN is non-invertible.The inverse recomputes the transformation from the unchanged channels and mel-spectrogram.
- Architecture: Invertible 1x1 convolutions mix information across channels before each affine coupling layer, and their Jacobian terms remain part of the loss.The convolution weights are initialized orthonormally to ensure invertibility.
- Multi-scale outputs: WaveGlow outputs two channels after every four coupling layers, then concatenates them with the final vectors to form z.Early outputs support information processing at multiple time scales and help gradients reach earlier layers.
- Inference: During inference, WaveGlow samples z from a Gaussian and runs the network inversely; a standard deviation of 0.6 was used after training with σ = √0.5.The inverse convolutions and affine transforms are guaranteed by the loss.
3. EXPERIMENTS
The experiments evaluate WaveGlow and comparison systems on LJ Speech using mel-spectrogram inputs, measuring subjective audio quality and inference speed. WaveGlow achieves high synthesis speed, while MOS differences are weak despite its highest score.
- Experimental setup: 13,100 LJ Speech clips from one speaker provide roughly 24 hours of training data recorded in a home environment.The recordings use a 22,050kHz sampling rate.
- Experimental setup: WaveGlow and WaveNet use mel-spectrograms, while Griffin-Lim uses the entire spectrogram and 60 frequency-to-time iterations.WaveGlow uses 80-bin mel-spectrograms; Griffin-Lim estimates missing phase iteratively.
- Experimental setup: 580,000 training iterations used randomly chosen 16,000-sample clips across 8 Nvidia GV100 GPUs.The WaveGlow network has 12 coupling layers and 12 invertible 1x1 convolutions; its learning rate was reduced after training appeared to plateau.
- Audio quality comparison: Approximately 1,000 MOS ratings show WaveGlow with the highest MOS, but all methods have similar scores and none reaches real-audio quality.The scores are reported with 95% confidence intervals, and differences are only weakly significant.
- Speed of inference comparison: 507kHz is the reported Griffin-Lim synthesis rate, whereas the compared WaveNet implementation runs at 0.11kHz.Griffin-Lim's figure uses 60 iterations, and WaveNet is described as significantly slower than real time.
- Speed of inference comparison: 520kHz is WaveGlow's approximate synthesis rate for a 10-second utterance on an NVIDIA V100 GPU.This unoptimized PyTorch implementation is slightly faster than the 500kHz reported for Parallel WaveNet; shorter utterances reduce speed per sample.
4. DISCUSSION
The discussion places WaveGlow among autoregressive and non-autoregressive speech-synthesis approaches. It emphasizes efficient synthesis with a simple, easily trained model while describing the complexity of competing non-autoregressive systems.
- Existing approaches: Autoregressive models capture long-term dependencies by conditioning future samples on previous samples, but their inference is computationally challenging.WaveNet is identified as the first such neural-network model discussed, with later models attempting to accelerate inference while retaining quality.
- Existing approaches: MCNN exceeds 5,000kHz but uses four hand-engineered losses and operates on the full spectrogram rather than reduced mel-spectrogram features.The discussion describes its training procedure as complicated and questions how its non-generative design would generate realistic audio.
- WaveGlow's approach: Flow-based models provide tractable likelihoods through invertibility, while WaveGlow combines Glow's flow approach with WaveNet's architectural insights.Parallel WaveNet and ClariNet also use flow-based models and can capture strong long-term dependencies in one individual pass.
- Conclusion: WaveGlow is presented as an efficient speech-synthesis model that is simple and easy to train, supporting potential deployment of high-quality audio synthesis.The discussion states this consequence without extending it beyond deployment support.