PaperlayerView exact revision on arXiv

Source-linked AI summary

Non-Autoregressive Neural Text-to-Speech

Kainan Peng, Wei Ping, Zhao Song, Kexin Zhao

arXiv:1905.08459v3cs.CLcs.LGcs.SDeess.AS

TL;DR

Autoregressive TTS pipelines can be slow because they generate spectrograms and waveforms sequentially. ParaNet uses a non-autoregressive text-to-spectrogram model with parallel neural vocoders, including a VAE-based IAF approach trained from scratch. It provides 46.7 times speed-up over its autoregressive counterpart while obtaining reasonably good speech quality, though parallel vocoders are generally less robust than WaveNet with a non-autoregressive front end.

  • Problem

    Autoregressive TTS pipelines remain slow at synthesis because waveform samples and spectrogram features are generated sequentially.

    Source: §1 · p. 1

  • Method

    ParaNet is a fully convolutional non-autoregressive text-to-spectrogram model, paired with parallel neural vocoders for feed-forward speech generation.

    Source: §1 · p. 1

  • Results

    46.7 times speed-up over its autoregressive counterpart at synthesis, with reasonably good speech quality.

    Source: §6 · p. 9

  • Takeaways & Limitations

    The results motivate further study of small-footprint, robust parallel neural vocoders for non-autoregressive acoustic models.

    Source: §6 · p. 9

    Parallel vocoders are generally less robust than WaveNet when the front-end acoustic model is non-autoregressive.

    Source: §6 · p. 9

Abstract

from arXiv · show

In this work, we propose ParaNet, a non-autoregressive seq2seq model that converts text to spectrogram. It is fully convolutional and brings 46.7 times speed-up over the lightweight Deep Voice 3 at synthesis, while obtaining reasonably good speech quality. ParaNet also produces stable alignment between text and speech on the challenging test sentences by iteratively improving the attention in a layer-by-layer manner. Furthermore, we build the parallel text-to-speech system and test various parallel neural vocoders, which can synthesize speech from text through a single feed-forward pass. We also explore a novel VAE-based approach to train the inverse autoregressive flow (IAF) based parallel vocoder from scratch, which avoids the need for distillation from a separately trained WaveNet as previous work.

1. Introduction

Neural TTS simplifies hand-engineered pipelines, but autoregressive spectrogram and waveform generation remains slow. ParaNet replaces the text-to-spectrogram stage with a fully convolutional non-autoregressive model and combines it with parallel vocoders for feed-forward speech synthesis.

1. Introduction

  • Autoregressive TTS can be slow because it generates waveform samples and spectrogram features sequentially, while existing pipelines retain autoregressive or recurrent components.End-to-end systems retain autoregressive spectrogram prediction; linguistic-feature systems rely on recurrent or autoregressive duration and F0 models.Source: §1 · p. 1
  • ParaNet runs 254.6 times faster than real-time and provides 46.7 times speed-up over autoregressive Deep Voice 3 while obtaining reasonably good speech quality.The comparison is at synthesis; speech quality is obtained using neural vocoders.Source: §1 · p. 1
  • ParaNet distills attention from an autoregressive text-to-spectrogram model and refines text-spectrogram alignment layer by layer, producing more stable attention on challenging sentences than Deep Voice 3.The authors attribute this stability to avoiding the discrepancy between teacher-forced training and autoregressive inference.Source: §1 · p. 1
  • Combining ParaNet with parallel neural vocoders yields a TTS system that generates speech from text in a single feed-forward pass.The investigated vocoders include distilled IAF and WaveGlow.Source: §1 · p. 1
  • WaveVAE trains an IAF-based parallel vocoder from scratch within a VAE framework.This alternative is introduced alongside the system's investigation of parallel vocoders.Source: §1 · pp. 1–2

2. Related work

Prior neural TTS systems simplify synthesis pipelines but are autoregressive, and parallel waveform generation does not necessarily remove sequential stages from the full system. The related work contrasts those approaches with non-autoregressive decoding and discusses flow-based vocoders and feed-forward VAE decoders.

2. Related work

  • The cited state-of-the-art neural TTS systems are based on autoregressive models, including attention-based seq2seq systems such as Deep Voice 3, Tacotron, and Char2Wav.Their seq2seq framework yields a simpler pipeline than traditional multi-stage systems.Source: §2 · p. 2
  • CNN-based autoregressive models allow parallel training but still generate outputs sequentially at synthesis, while some non-autoregressive machine-translation approaches use fertility, latent variables, or iterative refinement.The passage notes that TTS spectrograms may contain hundreds of time steps for a short text input.Source: §2 · p. 2
  • Compared with FastSpeech, ParaNet is fully convolutional and lightweight, has half the model parameters, uses a smaller training batch, and synthesizes faster.The stated training batch sizes are 16 for ParaNet and 64 for FastSpeech.Source: §2 · p. 2
  • Flow-based models have been used for parallel waveform synthesis, and this work uses a feed-forward IAF decoder in a VAE to enable parallel waveform synthesis.Earlier VAE work used autoregressive or recurrent decoders, which can be slow at synthesis.Source: §2 · p. 2

3. Text-to-spectrogram model

Section 3 presents an autoregressive text-to-spectrogram baseline and ParaNet, whose feed-forward decoder predicts full spectrograms in parallel. It then describes attention techniques for learning and refining text–spectrogram alignment.

3.1. Autoregressive architecture

  • The DV3-derived baseline encodes text with convolutions, uses a causal attention decoder for log-mel output, and applies a non-causal converter to predict log-linear spectrograms.Source: §3.1 · p. 2

3.2. Non-autoregressive architecture

  • ParaNet retains the encoder but predicts the entire log-mel sequence from encoder states in a feed-forward decoder, enabling parallel training and synthesis.Source: §3.2 · p. 3
  • Its non-causal decoder uses future context and predicts log-linear spectrograms too, so a separate non-causal converter is removed.Source: §3.2 · p. 3

3.3. Parallel attention mechanism

  • Because sequential location-sensitive attention conflicts with full parallelism, ParaNet stacks dot-product attention blocks that use encoder keys and values and layerwise queries to refine alignment.Source: §3.3 · pp. 3–4, Figure 3 · p. 4

3.3.1. ATTENTION DISTILLATION

  • Attention distillation trains ParaNet against a pretrained autoregressive teacher by minimizing average cross-entropy between their attention distributions, alongside spectrogram L1 losses.Source: §3.3.1 · p. 4

3.3.2. POSITIONAL ENCODING

  • Positional encodings added to keys and queries bias attention toward monotonic progression; the key rate roughly corresponds to speech speed.For ParaNet, the key rate at synthesis is 6.3/4, setting output length to 6.3/4 times the text length.Source: §3.3.2 · p. 4

3.3.3. ATTENTION MASKING

  • At synthesis, each query's softmax is restricted to a fixed encoder-key window around a position estimated from query time.The shared mask reduces repeating or skipping words and yields clearer pronunciations without preventing parallel synthesis.Source: §3.3.3 · p. 5

4. Parallel waveform model

The parallel waveform model converts ParaNet’s predicted mel spectrogram into raw waveform; this section reviews flow-based vocoders and introduces WaveVAE for training an IAF vocoder from scratch.

4.1. Flow-based waveform models

  • IAF supports parallel synthesis, but its autoregressive likelihood evaluation makes training slow; bipartite flows train from scratch but are less expressive and larger.WaveGlow has 87.88M parameters, compared with 2.17M for IAF.Source: §4.1 · p. 5

4.2. WaveVAE

  • WaveVAE jointly optimizes its encoder and decoder in a VAE framework, enabling IAF vocoder training from scratch without density distillation.The encoder is qφ(z|x, c), the decoder is pθ(x|z, c), and c denotes the mel-spectrogram conditioner.Source: §4.2 · p. 5

4.2.1. ENCODER

  • The WaveVAE encoder uses a Gaussian autoregressive WaveNet to map ground-truth audio to a same-length latent representation.Its mean and scale whiten the posterior, and the resulting qφ(z|x) allows parallel latent sampling.Source: §4.2.1 · p. 5

4.2.2. DECODER

  • The decoder applies stacked IAF transformations, whose shifting and scaling variables are modeled by a Gaussian WaveNet.The resulting distribution is Gaussian, and the waveform is sampled using its total mean and scale.Source: §4.2.2 · pp. 5–6

4.2.3. TRAINING OBJECTIVE

  • Training maximizes the VAE evidence lower bound, with the KL divergence computed in closed form because both distributions are Gaussian.KL weight is gradually annealed from 0 to 1 to avoid posterior collapse during early training.Source: §4.2.3 · p. 6
  • The objective adds STFT losses for encoder-based reconstruction and prior-based synthesis, with the latter intended to reduce the gap between reconstruction and synthesis.The STFT loss combines magnitude ℓ2 and log-magnitude ℓ1 losses; both coefficients are set to one.Source: §4.2.3 · p. 6

5. Experiment

The experiments compare ParaNet and WaveVAE across speech quality, synthesis speed, attention errors, and component ablations. They find faster synthesis and fewer attention errors for ParaNet than for Deep Voice 3, alongside vocoder-dependent quality differences.

5.1. Settings

  • Experiments used about 20 hours from one female speaker, downsampled from 48 kHz to 24 kHz, and compared ParaNet and DV3 with WaveNet, ClariNet, WaveVAE, and WaveGlow.ParaNet and DV3 were trained for 500K steps; vocoders were trained on randomly chosen 0.5-second audio clips.Source: §5.1 · p. 6

5.2. Results

  • MOS was 4.09 for ParaNet with WaveNet versus 4.01 for DV3 with WaveNet; ParaNet with ClariNet scored 3.62.WaveVAE scored worse than distilled IAF but could be trained from scratch; WaveGlow denoising helped DV3-conditioned output, but not ParaNet-conditioned output, whose MOS degraded seriously.Source: §5.2 · pp. 7–8, Table 2 · p. 7
  • 46.7 times speed-up over DV3 and 254.6 times faster than real-time; ParaNet averaged 0.024 seconds versus DV3's 1.12 seconds.The test utterances averaged 6.11 seconds; synthesis speed was measured on a 1080 Ti with FP32 arithmetic.Source: Table 2 · p. 7, §5.2 · p. 8
  • 12 vs. 37 attention errors without masking, and 6 vs. 8 with masking, favored ParaNet over DV3 on the 100-sentence test set.The table caption identifies the 17-layer ParaNet with attention mask as having the fewest total errors.Source: §5.2 · p. 8, Table 3 · p. 8

5.3. Ablation study

  • Without attention distillation, ParaNet learned no meaningful alignment and produced unintelligible noise; without positional encoding, it learned blurry alignments and failed to synthesize intelligible speech.Increasing decoder depth from 6 or 12 layers to the default 17 reduced attention errors with and without attention masking.Source: §5.3 · pp. 8–9

6. Conclusion

The conclusion presents ParaNet as a non-autoregressive text-to-spectrogram model for feed-forward neural TTS. It reports a 46.7 times synthesis speed-up over its autoregressive counterpart and notes that parallel vocoders are generally less robust than WaveNet when paired with a non-autoregressive acoustic model.

6. Conclusion

  • ParaNet provides reasonably good speech quality and 46.7 times speed-up over its autoregressive counterpart at synthesis, while parallel vocoders are generally less robust than WaveNet with a non-autoregressive acoustic model.The authors identify small-footprint, robust parallel vocoders as a future research direction.Source: §6 · p. 9

A. 15-Sentence Test Set

Appendix A lists the 15 sentences used to quantify inference speed-up; the list uses % to mark pauses.

A. 15-Sentence Test Set

B. 100-Sentence Test Set

Appendix B lists the 100 sentences used to quantify the attention-error results in Table 3, marking pauses with %.

B. 100-Sentence Test Set

Loading 1905.08459v3…