Source-linked AI summary
WaveFlow: A Compact Flow-based Model for Raw Audio
Wei Ping, Kainan Peng, Kexin Zhao, Zhao Song
TL;DR
Raw-audio synthesis needs high fidelity, but autoregressive models can be slow and bipartite flows can be large. WaveFlow combines direct maximum-likelihood training with autoregressive functions and dilated 2-D convolutions. Its 5.91M-parameter model synthesizes 22.05 kHz high-fidelity speech 42.60× faster than real-time.
Problem
Raw-audio synthesis faces a trade-off: autoregressive models generate sequentially and can be prohibitively slow, while bipartite flows need many parameters to reach comparable capacity.
Source: §1 · p. 1Method
Results
5.91M-parameter WaveFlow synthesizes 22.05 kHz high-fidelity speech (MOS: 4.32) 42.60× faster than real-time.
Source: §5.2 · p. 7Takeaways & Limitations
WaveFlow provides a compact option for production TTS, where small memory footprints are preferred, especially for on-device deployment.
Source: §1 · p. 2For bipartite flows that squeeze time samples onto the channel dimension, this operation may lose temporal order and yield constant-frequency noise.
Source: §2.2 · p. 3
Abstract
from arXiv · showhide
In this work, we propose WaveFlow, a small-footprint generative flow for raw audio, which is directly trained with maximum likelihood. It handles the long-range structure of 1-D waveform with a dilated 2-D convolutional architecture, while modeling the local variations using expressive autoregressive functions. WaveFlow provides a unified view of likelihood-based models for 1-D data, including WaveNet and WaveGlow as special cases. It generates high-fidelity speech as WaveNet, while synthesizing several orders of magnitude faster as it only requires a few sequential steps to generate very long waveforms with hundreds of thousands of time-steps. Furthermore, it can significantly reduce the likelihood gap that has existed between autoregressive models and flow-based models for efficient synthesis. Finally, our small-footprint WaveFlow has only 5.91M parameters, which is 15$\times$ smaller than WaveGlow. It can generate 22.05 kHz high-fidelity audio 42.6$\times$ faster than real-time (at a rate of 939.3 kHz) on a V100 GPU without engineered inference kernels.
1. Introduction
WaveFlow targets raw-audio synthesis that combines high fidelity with efficient generation, addressing the slowness of autoregressive models and the size of bipartite flows. It uses a compact flow trained directly with maximum likelihood.
1. Introduction
- WaveNet generates waveform samples sequentially and is prohibitively slow for speech synthesis, often requiring engineered kernels for real-time inference.Real-time inference is a requirement for most production text-to-speech systems.Source: §1 · p. 1
- Bipartite flows require more layers, larger hidden sizes, and many more parameters than autoregressive models to reach comparable capacity.WaveGlow and FloWaveNet are reported at 87.88M and 182.64M parameters, respectively, versus 4.57M for a typical 30-layer WaveNet.Source: §1 · p. 1
- 5.91M-parameter WaveFlow synthesizes 22.05 kHz high-fidelity speech (MOS: 4.32) more than 40× faster than real-time; WaveGlow needs 87.88M parameters for high-fidelity speech.The reported speed is on an Nvidia V100 GPU.Source: §1 · p. 2
- WaveFlow is directly trained with maximum likelihood, without the probability-density distillation and auxiliary losses used in other approaches, simplifying training and reducing development cost.It also avoids requiring a pretrained WaveNet teacher.Source: §1 · pp. 1–2
- WaveFlow squeezes waveform samples into a 2-D matrix, models adjacent samples autoregressively without losing temporal order, and uses dilated 2-D convolutions.The architecture is reported to have 15× fewer parameters and faster synthesis than WaveGlow.Source: §1 · p. 2
- WaveFlow unifies likelihood-based raw-audio models, with WaveNet and WaveGlow as special cases, and makes inference parallelism tradeable against model capacity.The authors report comparable likelihood and high-fidelity speech to WaveNet, with synthesis thousands of times faster.Source: §1 · p. 2
2. Flow-based generative models
Flow models transform a simple density into a complex data distribution through invertible mappings, using triangular Jacobians for tractable determinants. Autoregressive and bipartite transformations differ in their expressiveness and in which operations they parallelize.
2. Flow-based generative models
- Triangular Jacobians make flow-model determinants tractable, avoiding the O(n^3) cost of a general determinant.Flow models transform a simple density p(z) into a complex data distribution p(x) with an invertible mapping.Source: §2 · p. 2
2.1. Autoregressive transformation
- Autoregressive flows parallelize density evaluation but synthesize sequentially, whereas inverse autoregressive flows parallelize sampling but make density evaluation slow.In AF, each z_t depends only on x_≤t; IAF is used for parallel synthesis in Parallel WaveNet and ClariNet.Source: §2.1 · p. 2
2.2. Bipartite transformation
- Bipartite flows partition x into two groups, use a feed-forward network for shifts and scales, and allow both density evaluation and sampling in parallel.Their inverse mapping has a special triangular Jacobian.Source: §2.2 · pp. 2–3
- Table 1 compares sequential operations for density evaluation and sampling across flow models; increasing WaveFlow's squeezed height h raises capacity but also adds sampling steps.The caption defines n as waveform length and h as WaveFlow's squeezed height.Source: Table 1 · p. 3
- Channel-squeezing in WaveGlow and FloWaveNet can lose temporal order information, and the resulting audio can have constant-frequency noise.Source: §2.2 · p. 3
2.3. Connections
- Autoregressive transformations are more expressive: Figure 1 contrasts their greater number of nonlinear and linear dependencies with bipartite transformations.The paper describes bipartite flows as requiring more layers and larger hidden sizes to reach autoregressive-model capacity, such as measured by likelihood.Source: §2.3 · p. 3, Figure 1 · p. 3
3. WaveFlow
WaveFlow reshapes the waveform into a 2-D representation and uses autoregressive transformations with dilated 2-D convolutions. Height-wise permutations improve likelihood, with a split-and-reverse strategy outperforming simple reversal.
3.1. Definition
- WaveFlow squeezes adjacent waveform samples into columns of an h-row matrix and computes each latent element from its current sample and rows above it.This dependence gives a triangular Jacobian, enabling parallel likelihood evaluation and maximum-likelihood training.Source: §3.1 · pp. 3–4
- Unlike WaveNet, WaveFlow and WaveGlow can use future waveform samples; WaveNet is equivalent to a column-major autoregressive flow.When h > 2, WaveFlow's receptive field over squeezed inputs is strictly larger than WaveGlow's.Source: §3.1 · pp. 3–4, Figure 2 · p. 4
3.2. Implementation with dilated 2-D convolutions
- Dilated 2-D convolutions model the shifting and scaling functions; the height receptive field should cover h to avoid unnecessary conditional independence and lower likelihood.The convolutions are non-causal over width and causal over height; dilation choices determine the height receptive field.Source: §3.2 · p. 4
- WaveFlow becomes a Gaussian WaveNet when h equals waveform length, and a WaveGlow with two squeezed channels when h = 2 and height filter size is 1.The authors also implement a convolution queue to cache hidden states and speed autoregressive inference over height.Source: §3.2 · p. 4
3.3. Local conditioning for speech synthesis
- The mel spectrogram is upsampled to waveform length, reshaped to align with the waveform, and added as a bias at each layer after a 1 × 1 channel mapping.The reshaped conditioning input has dimensions c × h × w, where c is the number of input channels.Source: §3.3 · p. 5
3.4. Stacking multiple flows with permutations on height dimension
- Stacked flows map X = Z(n) through inverse transformations toward isotropic-Gaussian Z(0), and evaluate p(X) by the chain rule.For an eight-flow model, the tested permutations either reverse each height dimension or bipartition and reverse the lower four representations.Source: §3.4 · p. 5
- Height-wise permutations significantly improve likelihood; split-and-reverse strategy b outperforms full reversal a, attributed to diverse autoregressive orders.For speech synthesis, the conditioner must be permuted along the height dimension with the aligned latent representation.Source: §3.4 · p. 5
4. Related work
Prior work pairs autoregressive vocoders' high fidelity with slow sequential synthesis, while flow-based waveform models offer parallel synthesis but have been described as less expressive. WaveFlow is contrasted with Subscale WaveRNN in the number of synthesis steps for long waveforms.
4. Related work
- WaveFlow can generate a long waveform within, for example, 16 steps, whereas Subscale WaveRNN requires at least n/16 steps when producing up to 16 samples per step.Subscale WaveRNN folds a long waveform into a batch of shorter sequences.Source: §4 · p. 5
- WaveGlow and FloWaveNet use maximum-likelihood training and parallel waveform synthesis, but are described as less expressive than autoregressive models, with larger memory footprints and lower likelihood scores.The cited flow-based waveform models achieve fidelity comparable to autoregressive models.Source: §4 · p. 5
5. Experiment
The experiments compare likelihood, speech fidelity, synthesis speed, and text-to-speech performance. WaveFlow combines strong likelihood and speech quality with a small footprint and fast synthesis.
5. Experiment
- Core comparisons use 13,100 clips from the single-speaker, 24-hour LJ dataset at 22.05 kHz, with 80-band mel conditioning for WaveNet, WaveGlow, and WaveFlow.The evaluated likelihood-based models include WaveFlow, Gaussian WaveNet, WaveGlow, and autoregressive flow.Source: §5 · p. 6
5.1. Likelihood
- 5.023 vs. 5.026 test LL: 5.91M-parameter WaveFlow (k) matches 268.29M-parameter WaveGlow (g); WaveFlow (t) reaches 5.101 vs. WaveNet’s 5.059.WaveFlow (r) with 128 residual channels has test LL 5.055 vs. WaveNet’s 5.059.Source: §5.1 · p. 6, Table 5 · p. 7
- Increasing flow count improves LL; raising WaveFlow’s squeezed height h also raises LL but slows inference through more sequential steps, approaching AF in the limit.The authors describe this as a trade-off between model capacity and inference parallelism.Source: §5.1 · p. 6
- Modest h lets WaveFlow close the existing LL gap between autoregressive and efficiently sampled flow models, consistent with autoregressive strength in local signal structure.The paper reports a positive correlation between test likelihood and MOS for likelihood-based models and suggests LL may help with model selection.Source: §5.1 · p. 6, §5.2 · p. 7
5.2. Speech fidelity and synthesis speed
- 5.91M parameters let WaveFlow (64 channels) reach MOS 4.32 at 42.60× real-time; WaveFlow (256 channels) scores MOS 4.43 vs. WaveGlow’s 4.34 and matches WaveNet at 8.42× real-time.The large model uses 128 sequential steps to synthesize waveforms with hundreds of thousands of time-steps.Source: §5.2 · p. 7, Table 6 · p. 8
5.3. Text-to-Speech
- On 20-hour proprietary 24 kHz female-speaker TTS data, WaveFlow is described as a compelling neural vocoder with simple likelihood-based training, high-fidelity ultra-fast synthesis, and small memory footprint.The systems were conditioned on teacher-forced mel spectrograms predicted from text by Deep Voice 3.Source: §5.3 · p. 8
6. Discussion
The discussion contrasts the modeling strengths and limitations of autoregressive, convolutional, and likelihood-based approaches. WaveFlow combines short-range autoregressive modeling with a non-autoregressive convolutional architecture for long-range correlations.
6. Discussion
- Autoregressive models capture local waveform variation and deliver high fidelity, but are slow at synthesis and less efficient at modeling long-range correlations.The paper notes that WaveNet requires highly engineered kernels for real-time speech synthesis.Source: §6 · p. 8
- Non-autoregressive convolutions capture long-range structure and synthesize quickly, but can produce spurious high-frequency components that hurt audio fidelity.Source: §6 · p. 9
- WaveFlow combines short-range autoregressive functions for local variations with non-autoregressive convolutions for long-range correlations.The paper presents this as combining the strengths of both approaches.Source: §6 · p. 9
A. Squeezing time-domain samples on channel dimension may raise artifacts
The appendix discusses artifacts associated with squeezing waveform samples onto the channel dimension in WaveGlow. The reported constant-frequency noise depends on the squeezing size.
A. Squeezing time-domain samples on channel dimension may raise artifacts
- Squeezing time-domain samples on the channel dimension can lose temporal order and produce constant-frequency noise whose frequencies are directly related to squeezing size.Source: A. Squeezing time-domain samples on… · p. 11
- Figure 4 compares WaveGlow spectrograms for squeezing size 4 without early output and size 8 with early output, the latter being the repository default.Source: Figure 4 · p. 11