Source-linked AI summary
Wave-Tacotron: Spectrogram-free end-to-end text-to-speech synthesis
Ron J. Weiss, RJ Skerry-Ryan, Eric Battenberg, Soroosh Mariooryad, Diederik P. Kingma
TL;DR
Waveform-level text-to-speech has been difficult to model efficiently because speech samples have strong temporal dependencies, while conventional systems divide synthesis between text-to-feature prediction and waveform generation. Wave-Tacotron combines a Tacotron-like autoregressive decoder with a normalizing flow to generate waveform blocks directly, and the authors report high-quality speech generation without a separate vocoder; its blockwise design nevertheless retains sequential decoding.
Problem
Direct waveform generation from text has been elusive because strong temporal dependencies make sample-level autoregressive vocoders inefficient on parallel hardware.
Source: §1 · p. 1Method
Results
Wave-Tacotron directly generates high-quality speech waveforms from text, while its normalizing flow provides parallel generation capabilities within the autoregressive decoder structure.
Source: §4 · p. 4Takeaways & Limitations
The model generates waveform samples directly in a single model and can be trained by maximizing data likelihood without hand-designed intermediate-feature losses.
Source: §4 · p. 4The decoder still generates output blocks sequentially, disadvantaging Wave-Tacotron against parallel TTS unless its output step size can be made very large.
Source: §4 · p. 4
Abstract
from arXiv · showhide
We describe a sequence-to-sequence neural network which directly generates speech waveforms from text inputs. The architecture extends the Tacotron model by incorporating a normalizing flow into the autoregressive decoder loop. Output waveforms are modeled as a sequence of non-overlapping fixed-length blocks, each one containing hundreds of samples. The interdependencies of waveform samples within each block are modeled using the normalizing flow, enabling parallel training and synthesis. Longer-term dependencies are handled autoregressively by conditioning each flow on preceding blocks.This model can be optimized directly with maximum likelihood, with-out using intermediate, hand-designed features nor additional loss terms. Contemporary state-of-the-art text-to-speech (TTS) systems use a cascade of separately learned models: one (such as Tacotron) which generates intermediate features (such as spectrograms) from text, followed by a vocoder (such as WaveRNN) which generates waveform samples from the intermediate features. The proposed system, in contrast, does not use a fixed intermediate representation, and learns all parameters end-to-end. Experiments show that the proposed model generates speech with quality approaching a state-of-the-art neural TTS system, with significantly improved generation speed.
1. INTRODUCTION
Modern TTS often separates text-to-feature prediction from waveform vocoding, but direct waveform generation is difficult because of strong temporal dependencies and inefficient sample-by-sample generation. Wave-Tacotron combines text alignment with blockwise normalizing flows to generate waveforms directly, without spectrogram generation.
1. INTRODUCTION
- Recent TTS systems commonly predict intermediate audio features from text and then use a neural vocoder to produce waveform samples.Intermediate features may represent longer-term structure, while the vocoder supplies low-level signal detail.Source: §1 · p. 1
- The two-stage approach can complicate training and deployment because high quality generally requires fine-tuning or training the vocoder on synthesis-model outputs.Source: §1 · p. 1
- Direct text-to-waveform generation is difficult because waveform dependencies, including phase consistency, extend over time, while sample-level autoregressive generation is inefficient on parallel hardware.Source: §1 · p. 1
- Wave-Tacotron integrates a flow-based vocoder into a Tacotron-like, block-level autoregressive model conditioned on characters or phonemes.The text-attending decoder produces conditioning features for a normalizing flow that generates waveform blocks.Source: §1 · p. 1
- Fixed-length waveform blocks use flows to model within-block dependencies in parallel and condition autoregressively on previous blocks for longer-term dependencies.The model is reported to align text and generate high-fidelity speech without framing artifacts.Source: §1 · p. 1
- Unlike prior end-to-end waveform approaches [19] and [20], Wave-Tacotron avoids spectrogram generation and directly models time-domain waveforms by maximizing likelihood.The cited prior approaches rely on spectral losses and mel spectrograms for alignment.Source: §1 · p. 1
2. WAVE-TACOTRON MODEL
Wave-Tacotron combines a Tacotron-style text encoder and attention decoder with a normalizing flow that generates waveform blocks. Its flow models within-block structure while autoregressive conditioning links successive blocks.
2. WAVE-TACOTRON MODEL
- Wave-Tacotron couples a Tacotron-style attention decoder to a normalizing flow that emits non-overlapping waveform blocks rather than spectrogram frames.Source: §2 · p. 1, Figure 1 · p. 2
- The default block spans 40 ms, with K = 960 samples at 24 kHz; a decoder skip connection gives the flow preceding samples to avoid block-boundary discontinuities.K controls the trade-off between flow parallelizability and sample quality through autoregression.Source: §2 · p. 2
2.1. Flow
- The flow maps Gaussian noise to waveform segments and inverts target segments to Gaussian points with tractable density for likelihood training.It uses affine coupling layers for efficient density computation and sampling.Source: §2.1 · p. 2
- The multiscale flow squeezes sequences between stages, halving timesteps and doubling dimensions; conditioning features are upsampled and paired with position embeddings.The default configuration uses five stages with twelve flow steps each.Source: §2.1 · p. 2, Figure 2 · p. 2
- Waveform samples are quantized, dequantized with uniform noise, and pre-emphasized; generation uses a reduced prior temperature, with T = 0.7 as the default.The signal is de-emphasized at the model output; pre-emphasis improved subjective listening results but was not critical.Source: §2.1 · p. 2
2.2. Flow vocoder
- The flow-vocoder baseline generates waveforms from mel spectrograms with a highly parallelizable feed-forward architecture.It upsamples mel features to the flow frame rate and conditions the flow on them.Source: §2.2 · p. 3
3. EXPERIMENTS
Experiments compare Wave-Tacotron with alternative TTS systems on two single-speaker datasets, assessing speech quality, generation speed, and model variations. Wave-Tacotron improves on Griffin-Lim but trails Tacotron+WaveRNN in MOS, while its speed and ablations expose quality–efficiency trade-offs.
3. EXPERIMENTS
- Experiments use proprietary 39-hour and LJ 22-hour single-speaker datasets, comparing Wave-Tacotron with Tacotron+WaveRNN, Flowcoder, and Tacotron-PN, with MOS as the primary quality measure.Listening tests use native speakers with headphones; lower values are better for MCD, MSD, and CER, while higher MOS is better.Source: §3 · p. 3
- Wave-Tacotron performs substantially better than Griffin-Lim but not as well as Tacotron+WaveRNN in MOS; LJ shows the same ranking with a larger MOS gap.On the proprietary dataset, Wave-Tacotron has higher MCD and MSD than Tacotron+WaveRNN, but similar CER and only slightly lower MOS.Source: §3 · p. 3, Table 1 · p. 3, Table 2 · p. 4
- On LJ, Wave-Tacotron and Flowcoder have lower MSD, suggesting flows are more prone to fine-time artifacts; the authors conclude end-to-end training likely requires more data.Hyperparameters were tuned on the proprietary dataset.Source: §3 · p. 3, Table 2 · p. 4
3.1. Generation speed
- On TPU, Wave-Tacotron with R = 3 is about 17% slower than Tacotron + Flowcoder and about twice as fast as Tacotron-PN using 1000 Griffin-Lim iterations.Tacotron+WaveRNN is slightly slower than real-time on TPU; Wave-Tacotron's flow overhead is small on parallel hardware.Source: §3.1 · p. 4, Table 3 · p. 4
3.2. Ablations
- Removing pre-emphasis, conditioning-stack position embeddings, or the decoder skip connection hurts performance; removing the skip connection causes boundary clicks and raises MCD, MSD, and CER.The skip-connection variant was not evaluated in listening tests.Source: §3.2 · p. 4, Table 4 · p. 4
- Halving coupling-layer channels to 128 or reducing flow stages M or steps per stage N significantly reduces MOS.The ablations use a shallow two-layer decoder with phoneme inputs.Source: §3.2 · p. 4, Table 4 · p. 4
- Reducing R does not hurt MOS but slows generation; increasing R to 4 significantly reduces MOS through poor prosody and unnaturally fast speech.The authors suggest increasing decoder context or LSTM capacity as possible ways to mitigate this trade-off.Source: §3.2 · p. 4
4. DISCUSSION
Wave-Tacotron generates waveform speech directly from text by integrating a normalizing flow into the autoregressive Tacotron decoder. It avoids a separate vocoder and hand-designed intermediate features, but still generates blocks sequentially.
4. DISCUSSION
- Wave-Tacotron directly generates high-quality speech waveforms from text with one model, without a separate vocoder.Its hybrid structure combines attention-based TTS with a flow’s parallel generation capabilities.Source: §4 · p. 4
- Training maximizes data likelihood without complicated losses on hand-designed spectrogram or other mid-level features.Source: §4 · p. 4
- The decoder remains autoregressive across output blocks, so generation is sequential unless the block size becomes very large.Adapting the model to fully parallel TTS is left for future work.Source: §4 · p. 4
A. AUTOREGRESSIVE INPUT
A listening test compared the default reduction-factor setting with a variant that omitted reduction-factor truncation. The test found no preference between them.
A. AUTOREGRESSIVE INPUT
- The listening test compared default R = 3 against a no-truncation variant with K = 960 and R = 1 on proprietary phoneme-input data.Source: A. AUTOREGRESSIVE INPUT · p. 6
- The average preference score was 0.028 ± 0.126, indicating no preference between the models.Raters compared samples side by side on a scale from -3 to 3.Source: A. AUTOREGRESSIVE INPUT · p. 6
- Reduction-factor truncation means the decoder never sees the beginnings of output blocks at subsequent steps, but the test found no generation-quality impairment.Source: A. AUTOREGRESSIVE INPUT · p. 6
B. SAMPLE VARIABILITY
Wave-Tacotron samples show more pitch-track variation than spectrogram-based Tacotron, and that variation decreases at lower sampling temperatures. The authors suggest this variability could support speech-recognition data augmentation.
B. SAMPLE VARIABILITY
- Wave-Tacotron shows significantly more pitch-track variation across samples than spectrogram-based Tacotron, and lower sampling temperatures reduce that variation.The difference is visible in pitch trajectories across the utterance, including the first 0.7 seconds.Source: B. SAMPLE VARIABILITY · p. 6, Figure 3 · p. 6
- Figure 3 compares pitch tracks from 12 samples of the same text, with different systems in the top rows and Wave-Tacotron temperatures in the bottom rows.The tracks show variation in timing and pitch trajectories.Source: Figure 3 · p. 6, B. SAMPLE VARIABILITY · p. 6
- The authors attribute Wave-Tacotron’s richer prosodic variation to likelihood training, while noting that its high-dimensional noise inputs are difficult to interpret and control.Those noise inputs also entangle prosodic features with waveform details such as phase and precise timing.Source: B. SAMPLE VARIABILITY · p. 6