PaperlayerView exact revision on arXiv

Source-linked AI summary

Non-Attentive Tacotron: Robust and Controllable Neural TTS Synthesis Including Unsupervised Duration Modeling

Jonathan Shen, Ye Jia, Mike Chrzanowski, Yu Zhang, Isaac Elias, Heiga Zen, Yonghui Wu

arXiv:2010.04301v4cs.SDcs.CL

TL;DR

Attention-based autoregressive TTS can produce natural speech but is less robust, and small test sets may not capture its rare failures. Non-Attentive Tacotron replaces attention with explicit duration prediction and Gaussian upsampling, and uses a fine-grained VAE for duration modeling when target durations are scarce. The paper reports improved robustness over Tacotron 2, slightly better naturalness, and duration control at inference time.

  • Problem

    Attention-based autoregressive TTS can suffer rare synthesis failures that small handpicked test sets may not reflect, limiting robustness evaluation.

    Source: §1 · p. 1, §5 · p. 5

  • Method

    Non-Attentive Tacotron replaces attention with explicit duration prediction and Gaussian upsampling, and uses a fine-grained VAE for semi-supervised or unsupervised duration modeling.

    Source: §1 · p. 2, §4 · p. 5

  • Results

    Non-Attentive Tacotron significantly improves robustness over Tacotron 2 by UDR and word deletion rate, slightly outperforms it in naturalness, and achieves results almost as good as supervised training for unsupervised or semi-supervised duration modeling.

    Source: §7 · p. 9

  • Takeaways & Limitations

    The duration predictor supports utterance-wide and individual-word pacing control at inference time.

    Source: §7 · p. 9

    Semi-supervised and unsupervised models can occasionally produce unclear pronunciations, phoneme repetitions, or extra pauses, usually affecting only one or a few phonemes.

    Source: §6.3 · p. 8

Abstract

from arXiv · show

This paper presents Non-Attentive Tacotron based on the Tacotron 2 text-to-speech model, replacing the attention mechanism with an explicit duration predictor. This improves robustness significantly as measured by unaligned duration ratio and word deletion rate, two metrics introduced in this paper for large-scale robustness evaluation using a pre-trained speech recognition model. With the use of Gaussian upsampling, Non-Attentive Tacotron achieves a 5-scale mean opinion score for naturalness of 4.41, slightly outperforming Tacotron 2. The duration predictor enables both utterance-wide and per-phoneme control of duration at inference time. When accurate target durations are scarce or unavailable in the training data, we propose a method using a fine-grained variational auto-encoder to train the duration predictor in a semi-supervised or unsupervised manner, with results almost as good as supervised training.

1. Introduction

Attention-based autoregressive TTS can produce natural speech but is vulnerable to serious failures that small test sets may miss. Non-Attentive Tacotron replaces attention with duration prediction and upsampling, and adds unsupervised and semi-supervised duration modeling, robustness metrics, and duration controls.

1. Introduction

  • Attention-based autoregressive TTS can fail through early cut-off, repetition, skipping, long pauses, or babbling, and rare failures may be missed by small test sets.A single-frame stopping misprediction can cause serious failures, which may severely degrade customer experience.Source: §1 · p. 1
  • Non-Attentive Tacotron replaces Tacotron 2’s attention mechanism with duration prediction and upsampling to improve robustness while maintaining naturalness.The model combines the robust duration predictor with Tacotron 2’s autoregressive decoder.Source: §1 · p. 2
  • Unsupervised and semi-supervised duration modeling allows training with no or few duration annotations.Source: §1 · p. 2
  • The paper introduces unaligned duration ratio (UDR) and ASR word deletion rate (WDR) for automated large-scale evaluation of TTS robustness.Source: §1 · p. 2
  • Gaussian upsampling improves naturalness compared with vanilla repetition upsampling.Source: §1 · p. 2
  • The duration predictor supports global and fine-grained duration control at inference time while maintaining high synthesis quality.Source: §1 · p. 2

2. Related Works

Related work traces explicit duration representations from early neural parametric synthesis through attention-based end-to-end models and a renewed use of duration prediction in non-autoregressive TTS. Non-Attentive Tacotron is closest to DurIAN in combining duration prediction with an autoregressive decoder, while also allowing training without explicit duration labels.

2. Related Works

  • Early neural parametric synthesis and systems such as Deep Voice and CHiVE used explicit phoneme durations, while attention-based end-to-end models removed the need to represent durations explicitly.The cited attention-based examples include Char2Wav, Tacotron, Deep Voice 3, and Transformer TTS.Source: §2 · p. 2
  • Non-autoregressive TTS renewed explicit duration prediction, with examples including FastSpeech, AlignTTS, TalkNet, and JDI-T.Source: §2 · p. 2
  • Prior duration predictors obtain target durations from autoregressive models, CTC-based ASR, or external aligners, while AlignTTS trains without target durations using an alignment loss.Source: §2 · p. 2
  • Non-Attentive Tacotron is most similar to DurIAN because both combine a duration predictor with an autoregressive decoder, but this model can train without explicit duration labels.The paper describes its naturalness as similar to fully supervised training.Source: §2 · p. 2

3. Model

Non-Attentive Tacotron follows Tacotron 2’s encoder and autoregressive decoder but replaces attention-based alignment with duration-based upsampling. Gaussian upsampling forms weighted mixtures of encoder outputs, and the paper also describes externally aligned target durations and their limitations.

3. Model

  • The Tacotron 2-style autoregressive decoder predicts mel-spectrogram frames one at a time, using an upsampled encoder output instead of location-sensitive attention for alignment.At inference, the previous predicted frame is fed to the decoder; a post-net adds a residual to the preliminary spectrogram.Source: §3 · p. 3

3.1. Gaussian upsampling

  • Gaussian upsampling uses a learned range parameter for each token, predicted from encoder outputs concatenated with durations.Source: §3 · pp. 3–4

3. Model

  • The model combines duration-prediction loss with mel-spectrogram reconstruction loss, using L2 loss for durations and L1 + L2 loss for spectrograms.Source: §3 · p. 4

3.1. Gaussian upsampling

  • Gaussian upsampling places a Gaussian centered on each token’s duration-defined output segment and computes each frame as a weighted sum of encoder outputs.Its learned range parameter controls each token’s influence; unlike repetition upsampling, the method is fully differentiable, allowing spectrogram-loss gradients to reach the duration predictor.Source: §3.1 · p. 4, §3 · p. 4

3.2. Target Durations

  • Target durations come from an external flatstart-trained, speaker-dependent HMM-based aligner with a lexicon, but accurate alignment may be difficult with sparse data, poor recordings, or unclear pronunciations.The paper introduces semi-supervised and unsupervised duration modeling to address this problem.Source: §3.2 · p. 4

4. Unsupervised and semi-supervised duration modeling

The section proposes using a fine-grained VAE to derive duration-related token features from aligned target spectrograms, enabling duration prediction when target duration labels are scarce or unavailable. A naïve reconstruction-only approach using predicted durations does not produce satisfying naturalness.

4. Unsupervised and semi-supervised duration modeling

  • Predicted durations used for upsampling with only mel-spectrogram reconstruction loss do not produce satisfying naturalness.The naïve approach scales predicted per-token durations to match the target spectrogram length.Source: §4 · pp. 4–5
  • A fine-grained VAE models token-to-spectrogram alignment and extracts per-token latent features from the target mel-spectrogram for duration prediction.Token encoder outputs are aligned to the target spectrogram with attention; each latent feature is computed from the aligned context and token state.Source: §4 · p. 5
  • During training, durations are predicted from token encoder outputs and posterior latents; inference uses prior latents, sampled or set to the distribution mode.The FVAE’s internal attention mechanism is not used at inference.Source: §4 · p. 5
  • Semi-supervised training upsamples with target durations when available and predicted durations otherwise; duration loss applies only to examples with labels.Target durations may not be required with semi-supervised or unsupervised duration modeling.Source: §4 · pp. 4–5

5. Robustness evaluation

The section presents automated robustness evaluation using ASR and forced alignment on large evaluation sets, addressing limitations of handpicked hard cases. It measures over-generation with UDR and under-generation with WDR, while noting that ASR errors make these metrics upper bounds on actual TTS failures.

5. Robustness evaluation

  • Automated evaluation uses ASR and forced alignment on synthesized speech against verbalized text to measure over- and under-generation.The section frames this as a large-evaluation-set alternative to handpicked hard cases, which may be biased toward a system’s weaknesses and encourage overfitting to the evaluation set.Source: §5 · p. 5
  • UDR is the share of output duration in long (>1 second) unaligned segments; failed alignment is assigned a UDR of 100%.Long unaligned segments can reflect over-generation, including long pauses, babbling, repetitions, or failure to stop; short unaligned segments are ignored.Source: §5 · p. 5
  • WDR is the deletion-error portion of WER and reflects under-generation such as early cutoff or skipped words.Source: §5 · p. 5
  • ASR errors make these metrics upper bounds on actual TTS failures.Source: §5 · p. 5

6. Experiments

The experiments evaluate naturalness, pace control, duration modeling with limited annotations, robustness at scale, and a non-autoregressive decoder. Gaussian upsampling matched Tacotron 2 (GMMA) in naturalness, while robustness and decoder comparisons favored the supervised Non-Attentive Tacotron and its autoregressive decoder, respectively.

6. Experiments

  • The experiments used 66 speakers across four English accents, and naturalness tests evaluated 10 US English speakers with 3 to 47 hours of training data each.Tacotron 2 comparisons included location-sensitive and GMM attention; all experiments used a WaveRNN vocoder.Source: §6 · p. 6, §6.1 · p. 6

6.1. Naturalness

  • Non-Attentive Tacotron with Gaussian upsampling matched Tacotron 2 (GMMA) in naturalness, and both were close to ground-truth audio; vanilla repetition upsampling was significantly less natural.A preference test between Non-Attentive Tacotron and Tacotron 2 (GMMA) further confirmed the naturalness result.Source: §6.1 · pp. 6–7

6.2. Pace control

  • With pace between 0.8× and 1.25×, WER was hardly impacted, while subjective MOS decreased rapidly when pace changed significantly.Raters often attributed lower scores to pace being too slow or too fast; WER was significantly worse at 1.5× normal, partly because the ASR model was not optimized for such fast speech.Source: §6.2 · p. 7

6.3. Unsupervised and semi-supervised duration modeling

  • Without target durations, the FVAE and duration predictor produced alignments close to those computed from target durations.With the FVAE, semi-supervised and unsupervised naturalness was very close to supervised naturalness despite higher duration prediction errors; without the FVAE, naturalness degraded significantly.Source: §6.3 · p. 7, Figure 4 · p. 7
  • Semi-supervised and unsupervised samples had a small chance of slight errors absent from supervised samples, including unclear pronunciations, phoneme repetitions, or extra pauses.These errors were reported as less severe than similar Tacotron 2 errors, usually affecting only one or a few phonemes.Source: §6.3 · pp. 7–8

6.4. Robustness

  • Non-Attentive Tacotron had significantly less over-generation than Tacotron 2 (GMMA), including when duration modeling was semi-supervised or unsupervised; those variants were significantly worse than the supervised model in robustness.Semi-supervised and unsupervised models still performed significantly better on over-generation than Tacotron 2 and about the same on under-generation.Source: §6.4 · pp. 8–9, Table 4 · p. 8
  • Non-Attentive Tacotron was significantly less sensitive than Tacotron 2 to data preprocessing, which for Tacotron 2 required consistent silence trimming and filtering utterances with long pauses.Source: §6.4 · p. 9

6.5. Non-autoregressive decoder

  • Non-autoregressive decoders produced significantly less natural audio than the autoregressive decoder, and the results indicate the autoregressive decoder is critical to Non-Attentive Tacotron's performance.Larger non-autoregressive decoders performed significantly better than the 256×4 decoder.Source: §6.5 · p. 9, Table 5 · p. 9

7. Conclusions

Non-Attentive Tacotron improves robustness over Tacotron 2 while slightly outperforming it in naturalness. Its duration predictor also supports pacing control and duration modeling with scarce or unavailable annotations.

7. Conclusions

  • The model significantly improves robustness over Tacotron 2 on unaligned duration ratio and word deletion rate while slightly outperforming it in naturalness.The model replaces Tacotron 2’s attention mechanism with an explicit duration predictor and Gaussian upsampling.Source: §7 · p. 9
  • A fine-grained variational auto-encoder supports unsupervised or semi-supervised duration modeling when accurate target durations are scarce or unavailable.Results are almost as good as supervised training.Source: §7 · p. 9
  • The duration predictor enables control over the pacing of the entire utterance and individual words.Source: §7 · p. 9

A. Model Parameters

The section identifies a table of model parameters but provides no parameter values in the supplied passage.

A. Model Parameters

  • Table 6 presents model parameters, but the supplied passage does not state their values.Source: Table 6 · p. 13

B. WER breakdowns in the robustness evaluation

The section identifies a WER breakdown table for the robustness evaluation and defines deletion rate as the WDR reported in Table 4.

B. WER breakdowns in the robustness evaluation

  • Table 7 presents WER breakdowns for the robustness evaluation; its deletion rate (del) is the WDR in Table 4.Source: Table 7 · p. 14

C. Effectiveness of a learned σ in the Gaussian upsampling

The preference test reports a slight perceived benefit for a learned range parameter over fixed σ, while avoiding another dataset-dependent hyperparameter to tune.

C. Effectiveness of a learned σ in the Gaussian upsampling

D. Classification of some TTS models

Table 9 classifies selected TTS models by autoregressive versus feed-forward architecture, model family, and attention- versus duration-based design.

D. Classification of some TTS models

  • Table 9 groups some TTS models by autoregressive or feed-forward design, RNN, Transformer, or fully convolutional architecture, and attention- or duration-based approach.The supplied passage identifies these classification dimensions but does not provide the models’ individual classifications.Source: Table 9 · p. 14, D. Classification of some TTS models · p. 14
Loading 2010.04301v4…