PaperlayerView exact revision on arXiv

Source-linked AI summary

FastSpeech: Fast, Robust and Controllable Text to Speech

Yi Ren, Yangjun Ruan, Xu Tan, Tao Qin, Sheng Zhao, Zhou Zhao, Tie-Yan Liu

arXiv:1905.09263v5cs.CLcs.LGcs.SDeess.AS

TL;DR

Autoregressive TTS is slow and difficult to control, and can produce skipped or repeated words. FastSpeech predicts phoneme durations to expand phoneme sequences for parallel mel-spectrogram generation; on LJSpeech it nearly matches speech quality while accelerating synthesis and supporting smooth speed adjustment.

  • Problem

    Autoregressive TTS generates long mel-spectrogram sequences slowly and makes direct voice-speed and prosody control difficult, while wrong alignments can lead to skipped or repeated words.

    Source: §1 · p. 2

  • Method

    FastSpeech predicts phoneme durations and uses a length regulator to expand phoneme representations to mel-spectrogram length for parallel generation.

    Source: §1 · p. 2

  • Results

    270x faster mel-spectrogram generation and 38x faster end-to-end speech synthesis than autoregressive Transformer TTS, with near-matched speech quality and nearly eliminated word skipping and repeating.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Lengthening or shortening phoneme durations adjusts voice speed, and adding breaks between adjacent phonemes controls part of prosody.

    Source: §1 · p. 2

Abstract

from arXiv · show

Neural network based end-to-end text to speech (TTS) has significantly improved the quality of synthesized speech. Prominent methods (e.g., Tacotron 2) usually first generate mel-spectrogram from text, and then synthesize speech from the mel-spectrogram using vocoder such as WaveNet. Compared with traditional concatenative and statistical parametric approaches, neural network based end-to-end models suffer from slow inference speed, and the synthesized speech is usually not robust (i.e., some words are skipped or repeated) and lack of controllability (voice speed or prosody control). In this work, we propose a novel feed-forward network based on Transformer to generate mel-spectrogram in parallel for TTS. Specifically, we extract attention alignments from an encoder-decoder based teacher model for phoneme duration prediction, which is used by a length regulator to expand the source phoneme sequence to match the length of the target mel-spectrogram sequence for parallel mel-spectrogram generation. Experiments on the LJSpeech dataset show that our parallel model matches autoregressive models in terms of speech quality, nearly eliminates the problem of word skipping and repeating in particularly hard cases, and can adjust voice speed smoothly. Most importantly, compared with autoregressive Transformer TTS, our model speeds up mel-spectrogram generation by 270x and the end-to-end speech synthesis by 38x. Therefore, we call our model FastSpeech.

1 Introduction

FastSpeech targets the slow inference, skipped or repeated words, and limited voice control of autoregressive TTS with parallel mel-spectrogram generation. On LJSpeech, it nearly matches autoregressive Transformer TTS in speech quality while substantially speeding synthesis and enabling smooth speed adjustment.

1 Introduction

  • Autoregressive TTS faces slow inference because it conditions each mel-spectrogram on previously generated frames across sequences hundreds or thousands long.Source: §1 · p. 2
  • Autoregressive models make direct voice-speed and prosody control difficult because they generate mel-spectrograms sequentially without explicitly using text-speech alignments.Source: §1 · p. 2
  • A phoneme duration predictor and length regulator expand phoneme representations to the target mel-spectrogram length, enabling parallel generation with hard phoneme-to-spectrogram alignments.The hard alignments reduce error propagation and wrong attention alignments, lowering skipped and repeated words.Source: §1 · p. 2
  • Lengthening or shortening phoneme durations adjusts voice speed, while inserting breaks between adjacent phonemes controls part of prosody.Source: §1 · p. 2
  • 270x faster mel-spectrogram generation and 38x faster end-to-end speech synthesis than autoregressive Transformer TTS, with near-matched speech quality and nearly eliminated word skipping and repeating.The reported experiments use the LJSpeech dataset.Source: §1 · p. 2

2 Background

The background describes neural TTS as converting text into acoustic features and then audio, while noting that most systems generate mel-spectrograms autoregressively. It situates FastSpeech among sequence-to-sequence and non-autoregressive approaches, distinguishing its parallel mel generation from parallel vocoders.

2 Background

  • ClariNet is fully end-to-end, but it still generates mel-spectrograms autoregressively before synthesizing speech.Source: §2 · p. 2
  • Neural end-to-end TTS converts text into acoustic features such as mel-spectrograms and then audio; most systems generate mel-spectrograms autoregressively.The passage places neural methods after concatenative and statistical parametric synthesis and says end-to-end speech quality is close to human parity.Source: §2 · p. 3
  • Sequence-to-sequence learning typically uses an encoder for source representations and a decoder that predicts target elements conditioned on those representations and preceding elements, with attention selecting source representations.Source: §2 · p. 3
  • FastSpeech replaces the conventional encoder-attention-decoder framework with a feed-forward network that generates the sequence in parallel.Source: §2 · p. 3
  • FastSpeech addresses TTS robustness and controllability as well as speed; unlike parallel vocoders, it generates mel-spectrograms in parallel rather than relying on autoregressive mel generation.The passage also contrasts it with a concurrent parallel-mel approach that retains attention and does not fully resolve skipped and repeated words.Source: §2 · p. 3

3 FastSpeech

FastSpeech uses a feed-forward Transformer to generate mel-spectrograms in parallel, bridging the shorter phoneme sequence and longer spectrogram sequence with a length regulator. A duration predictor is trained from alignments extracted from an autoregressive teacher and supplies durations for inference.

3.1 Feed-Forward Transformer

  • FastSpeech stacks feed-forward Transformer blocks on phoneme and mel-spectrogram sides, with a length regulator between them for parallel generation.Each block combines self-attention with a two-layer 1D convolutional network.Source: §3.1 · p. 3, §3 · p. 3, Figure 1 · p. 4

3.2 Length Regulator

  • The length regulator repeats each phoneme hidden state according to its duration, making the expanded sequence match the mel-spectrogram length.A scaling factor α lengthens or shortens durations to control voice speed; changing space-character duration can adjust breaks between words.Source: §3.2 · p. 4

3.3 Duration Predictor

  • A two-layer 1D-convolutional duration predictor estimates each phoneme’s duration with MSE loss and is used during inference.It predicts durations in the logarithmic domain; during training, extracted teacher durations are used instead.Source: §3.3 · pp. 4–5
  • Teacher attention alignments provide duration targets: the method selects the attention head with the highest focus rate, then counts mel frames assigned to each phoneme.The focus rate measures how closely an attention head follows a diagonal alignment.Source: §3.3 · p. 5

4 Experimental Setup

Experiments use LJSpeech, with a separate set of particularly hard sentences for robustness evaluation. The setup trains an autoregressive Transformer teacher and FastSpeech, then uses pretrained WaveGlow to synthesize audio from FastSpeech mel-spectrograms.

4.1 Datasets

  • The LJSpeech split contains 12,500 training, 300 validation, and 300 test samples; robustness is also evaluated on 50 particularly hard sentences.The dataset has 13,100 English audio clips totaling approximately 24 hours.Source: §4.1 · p. 5

4.2 Model Configuration

  • FastSpeech uses six FFT blocks on each side, while its autoregressive Transformer teacher has six encoder and six decoder layers and serves duration extraction and sequence-level distillation.The teacher has a number of parameters similar to FastSpeech.Source: §4.2 · p. 5

4.3 Training and Inference

  • Both models are trained for about 80k steps on four NVIDIA V100 GPUs, and pretrained WaveGlow converts FastSpeech mel-spectrograms into audio.Teacher attention alignments train the duration predictor, and teacher-generated mel-spectrograms provide FastSpeech training pairs.Source: §4.3 · p. 6

5 Results

On LJSpeech, FastSpeech nearly matches Transformer TTS in audio quality while speeding inference, improving robustness on hard sentences, and enabling voice-speed and prosody control.

Audio Quality

  • FastSpeech nearly matches Transformer TTS and Tacotron 2 in MOS audio quality.The evaluation reports MOS with 95% confidence intervals.Source: Table 1 · p. 6, §5 · p. 6

Inference Speedup

  • FastSpeech achieves 269.40x speedup for mel-spectrogram generation and 38.30x speedup for audio generation versus Transformer TTS.The latency evaluation uses systems with similar parameter counts; end-to-end audio generation uses WaveGlow.Source: §5 · p. 6, Table 2 · p. 7

Robustness

  • On 50 particularly hard sentences, Transformer TTS has a 34% error rate, while FastSpeech effectively eliminates word repeating and skipping.Each kind of word error is counted at most once per sentence.Source: Table 3 · p. 7, §5 · p. 7

Voice Speed

  • FastSpeech adjusts voice speed smoothly from 0.5x to 1.5x while maintaining stable, almost unchanged pitch.The comparison shows mel-spectrograms for the same input text at 1.5x, 1.0x, and 0.5x speed.Source: §5 · pp. 7–8, Figure 3 · p. 8

Breaks Between Words

  • Lengthening space-character durations adds breaks between words to improve prosody.The example adds breaks after “deeply” and “especially.”Source: §5 · p. 8, Figure 4 · p. 8

Ablation Study

  • Replacing 1D convolution with a fully connected layer results in -0.113 CMOS, while removing sequence-level knowledge distillation results in -0.325 CMOS.Both are component ablations evaluated with CMOS.Source: Table 4 · p. 8, §5 · p. 8

6 Conclusions

FastSpeech generates mel-spectrograms in parallel with a feed-forward network, length regulator, and duration predictor. On LJSpeech it nearly matches Transformer TTS quality while improving speed, robustness, and voice-speed control; future work targets broader settings and parallel-vocoder training.

Conclusions

  • FastSpeech uses a feed-forward network to generate mel-spectrograms in parallel, with feed-forward Transformer blocks, a length regulator, and a duration predictor.Source: §6 · p. 9
  • On LJSpeech, FastSpeech nearly matches Transformer TTS quality, speeds mel-spectrogram generation by 270x and end-to-end synthesis by 38x, nearly eliminates word skipping and repeating, and supports 0.5x–1.5x voice speed.Source: §6 · p. 9

Future Work

  • Future work aims to improve speech quality, apply FastSpeech to multi-speaker and low-resource settings, and train it jointly with a parallel neural vocoder.Joint training is intended to make synthesis fully end-to-end and parallel.Source: §6 · p. 9

A Model Hyperparameters

Table 5 identifies the hyperparameter groups used for Transformer TTS and FastSpeech.

Hyperparameters

  • Table 5 lists hyperparameters for Transformer TTS and FastSpeech.Source: Table 5 · p. 11
  • Transformer TTS hyperparameters are labeled Encoder and Decoder.Source: Table 5 · p. 11
  • FastSpeech hyperparameters are labeled Phoneme-Side and Mel-Side FFT.Source: Table 5 · p. 11

B 50 Particularly Hard Sentences

The appendix lists the 50 particularly hard sentences used in the robustness evaluation, including examples with unusual strings and technical text.

50 Particularly Hard Sentences

Loading 1905.09263v5…