PaperlayerView exact revision on arXiv

Source-linked AI summary

End-to-End Adversarial Text-to-Speech

Jeff Donahue, Sander Dieleman, Mikołaj Bińkowski, Erich Elsen, Karen Simonyan

arXiv:2006.03575v3cs.SDcs.LGeess.AS

TL;DR

The paper addresses the complexity and supervision requirements of multi-stage TTS pipelines by learning speech synthesis directly from normalised text or phonemes. It introduces a feed-forward adversarial model with differentiable duration-based alignment and soft dynamic time warping, achieving a mean opinion score of 4.083 on a 5-point scale. The result approaches state-of-the-art systems that use multi-stage training or additional supervision.

  • Problem

    Typical TTS pipelines use independently designed or trained stages and may require stage-specific supervision, motivating end-to-end synthesis from text or phonemes.

  • Method

    A feed-forward generator predicts token durations for differentiable alignment and combines adversarial feedback with prediction losses, including soft dynamic time warping.

  • Results

    4.083 mean opinion score on a 5-point scale, approaching the state of the art from models trained using richer supervisory signals.

  • Takeaways & Limitations

    The model produces high-fidelity natural-sounding speech from normalised text or phonemes while remaining efficient in training and inference.

  • Takeaways & Limitations

    The approach still has a fidelity gap relative to state-of-the-art systems and relies on a separate fixed system for text normalisation and phonemisation.

Abstract

from arXiv · show

Modern text-to-speech synthesis pipelines typically involve multiple processing stages, each of which is designed or learnt independently from the rest. In this work, we take on the challenging task of learning to synthesise speech from normalised text or phonemes in an end-to-end manner, resulting in models which operate directly on character or phoneme input sequences and produce raw speech audio outputs. Our proposed generator is feed-forward and thus efficient for both training and inference, using a differentiable alignment scheme based on token length prediction. It learns to produce high fidelity audio through a combination of adversarial feedback and prediction losses constraining the generated audio to roughly match the ground truth in terms of its total duration and mel-spectrogram. To allow the model to capture temporal variation in the generated audio, we employ soft dynamic time warping in the spectrogram-based prediction loss. The resulting model achieves a mean opinion score exceeding 4 on a 5 point scale, which is comparable to the state-of-the-art models relying on multi-stage training and additional supervision.

1 INTRODUCTION

EATS targets end-to-end speech synthesis from character or phoneme sequences, replacing independently trained TTS stages with a differentiable feed-forward system. Its aligner and dynamic time-warping losses support high-fidelity speech, achieving a mean opinion score of 4.083.

  • Motivation: EATS simplifies TTS by learning speech synthesis from text or phonemes in an end-to-end manner.Typical pipelines independently design or train stages such as text processing, linguistic featurisation, mel-spectrogram synthesis, and waveform synthesis.
  • Architecture: The system uses a feed-forward aligner that predicts input-token durations and produces an audio-aligned representation.The aligner operates at 200 Hz, while the decoder upsamples its features through 1D convolutions to generate 24 kHz waveforms.
  • Training: Dynamic time warping-based prediction losses enforce conditioning alignment while allowing variability in human speech timing.This complements adversarial feedback and avoids requiring exactly aligned generated and target spectrograms.
  • Results: 4.083 mean opinion score demonstrates natural-sounding speech approaching systems trained with richer supervisory signals.The score is reported on a 5-point scale.

2 METHOD

The method learns speech synthesis nearly end-to-end with a feed-forward aligner-decoder generator, adversarial feedback, and domain-specific prediction losses. Its alignment predicts token lengths monotonically and uses soft dynamic time warping to accommodate timing variation.

  • Generator architecture: The generator maps characters or phonemes to raw 24 kHz audio through an aligner producing 200 Hz representations and a convolutional decoder.The aligner replaces earlier pipeline stages with learned aligned features before waveform upsampling.
  • Adversarial training: Adversarial loss supports efficient feed-forward training and inference while encouraging realistic speech in the strongly conditioned text-to-speech setting.The generator is trained with adversarial feedback paired with a discriminator hinge objective.
  • Monotonic alignment: Token lengths are predicted individually and cumulatively converted into positions, implicitly enforcing monotonic alignment while allowing sequence-length generalisation.Dilated convolutions provide broad receptive fields so tokens such as punctuation can influence the entire utterance.
  • Training windows: Training uses randomly extracted 2-second windows from utterances ranging from about 1 to 20 seconds, reducing memory requirements for 24 kHz audio.The aligner produces a 200 Hz representation for each training window before decoding.
  • Prediction losses: Adversarial feedback alone does not learn alignment reliably, so an explicit L1 mel-spectrogram prediction loss guides training and avoids conditional discriminators.Spectrogram-domain comparison is more invariant to perceptually insignificant phase differences than time-domain comparison.
  • Dynamic time warping: Soft dynamic time warping replaces rigid spectrogram alignment with a differentiable weighted aggregation over feasible paths, enabling stochastic timing while preserving gradient propagation.The method sets τ = 0.01 and λpred = 1.0; the resulting relaxed loss cooperates with adversarial training.

3 RELATED WORK

Prior TTS work commonly separates speech generation into sequential stages and often relies on aligned intermediate supervision. EATS instead uses a single feed-forward, end-to-end network that learns alignment from character or phoneme sequences while using dynamic time warping to accommodate timing variation.

  • Prior multi-stage approaches: Most TTS systems divide generation into temporally aligned intermediate representations and separately address spectrogram generation or waveform synthesis.These modular pipelines make the task more manageable but separate stages of the generation process.
  • Prior multi-stage approaches: Autoregressive TTS models have sequential output dependencies, making them impractical for real-time use without additional engineering.Flow-based models provide a feed-forward alternative for fast inference, but typically condition waveform generation on spectrograms or linguistic features.
  • EATS distinction: EATS is a single feed-forward neural network trained end-to-end in one stage from character or phoneme sequences, without auxiliary alignment supervision or teacher forcing.This differs from approaches such as Char2wav, which requires vocoder-feature pre-training for intermediate supervision.
  • EATS distinction: Soft dynamic time warping serves as a differentiable spectrogram prediction loss that relaxes alignment constraints and permits variability in human speech timing.The paper distinguishes this use from monotonic alignment search, which optimizes text-to-latent alignment.

4 EVALUATION

The evaluation measures speech naturalness with human MOS ratings on held-out conditioning sequences using professional-voice recordings. EATS reaches 4.083 MOS, while ablations show contributions from adversarial feedback, dynamic time warping, phoneme inputs, monotonic interpolation, and multi-speaker training.

  • Evaluation setup: Mean Opinion Score (MOS) is the primary metric, averaging 1–5 naturalness ratings from human raters across 1000 held-out conditioning sequences.Fréchet DeepSpeech Distance is additionally reported in an appendix.
  • Evaluation setup: The private training dataset contains 260.49 hours of 24 kHz recordings from 69 North American English-speaking professional voices.Clips are full sentences ranging from less than 1 to 20 seconds, with uneven amounts of data per speaker.
  • Main results: 4.083 MOS is achieved by the base multi-speaker EATS model, approaching prior reported MOS values while using substantially less supervision.Direct comparison is limited because datasets differ, except for the matched GAN-TTS evaluation setting.
  • Ablations: Removing raw-window discriminators reduces MOS to 2.526, while removing dynamic time warping reduces MOS to 3.559 and causes warbling and unnatural phoneme lengths.Removing the mel-spectrogram discriminator or all discriminators also produces substantially worse MOS than the full model.
  • Ablations: Character-only inputs achieve 3.423 MOS, attention-based alignment achieves 3.551 MOS, and single-speaker training achieves 3.829 MOS.The reported differences are associated with mispronunciations and unusual stress, poor generalisation to long utterances, and the benefit of multi-speaker data, respectively.
  • Speaker analysis: MOS generally improves with more training data across the four most prolific speakers, although the relationship is imperfect.Speaker #3 obtains the highest MOS despite having only the third-largest amount of training data.

5 DISCUSSION

The authors report that EATS learns speech from normalized text or phonemes with weak supervision and avoids autoregressive sampling and teacher forcing. They also identify remaining fidelity and preprocessing boundaries.

  • EATS learns from normalized text or phonemes paired with corresponding speech audio, rather than requiring stronger intermediate supervision.
  • The feed-forward system avoids autoregressive sampling, teacher forcing, exposure bias, and reduced inference parallelism.
  • The method still has a fidelity gap relative to state-of-the-art systems using multi-stage training pipelines or additional supervision.
  • The current approach leaves text normalisation and phonemisation to a separate fixed system rather than operating on unnormalised raw text.

A HYPERPARAMETERS AND OTHER DETAILS

The section gives training details and reports fast batched inference for EATS across TPU, GPU, and CPU hardware. GPU and TPU generation operate substantially faster than real time.

  • Models are trained for 5 · 10^5 steps with alternating discriminator and generator updates, minibatch size 1024, Adam, and cosine learning-rate decay.The learning rate starts at 10^-3 and decays to 0 at step 500K.
  • Inference speed: 206.8× realtime is achieved on a Google Cloud NVIDIA V100 GPU for batched speech generation.Generating 2400 seconds of speech takes 11.60 seconds on average.
  • Inference speed: 157.2× realtime per TPU chip and 8.52× realtime on a CPU are reported for EATS inference.The TPU figure is 78.6× realtime per core.
  • Figure 3 provides pseudocode for the EATS aligner described in Section 2.1.

C SPECTROGRAM DISCRIMINATOR ARCHITECTURE

The appendix documents the EATS aligner and spectrogram discriminator implementation, including token and speaker conditioning, masked dilated convolutions, and predicted token lengths.

  • Spectrogram discriminator: The spectrogram discriminator receives 47 × 80 × 1 inputs derived from 2-second, 24 kHz waveforms, using a BigGAN-deep-inspired architecture with selected blocks removed.
  • EATS aligner: Speaker embeddings and 128-dimensional Gaussian noise are concatenated to form class-conditioning inputs for conditional batch normalization.
  • EATS aligner: A masked stack of 10 blocks with six convolutions per block produces unaligned features while respecting valid token positions.
  • EATS aligner: The aligner predicts nonnegative token lengths, obtains token centers by cumulative summation, and derives the total aligned length from the final valid token.
  • Figure 3 is identified as pseudocode for the proposed EATS aligner, while Figure 4 is identified as TensorFlow code for mel-spectrogram computation.

D MEL-SPECTROGRAM COMPUTATION

The section documents mel-spectrogram processing and soft dynamic time warping used in prediction losses. It also describes phonemization choices and notes that the DTW implementation is quadratic.

  • Mel-spectrogram computation: Mel-spectrogram computation feeds both the spectrogram discriminator and spectrogram prediction loss, with jitter enabled only for real spectrograms in prediction losses.
  • Dynamic time warping: Soft dynamic time warping relaxes the assumption that generated and ground-truth spectrograms are exactly aligned by finding a minimum-cost alignment path.
  • Dynamic time warping: The current DTW implementation has quadratic complexity, although Itakura or Sakoe-Chiba bands could make it more efficient.
  • Phonemization: Phonemizer with the espeak backend produces IPA sequences, retaining stress marks and punctuation while removing spurious whitespace.
  • Phonemization: Rare phonemizer symbols are replaced with more frequent symbols, resulting in 51 distinct symbols.

G TRANSFORMER-BASED ATTENTION ALIGNER BASELINE

The paper evaluates a transformer-based attention aligner baseline against the proposed monotonic interpolation-based aligner. The figures examine how token positions and predicted sequence lengths vary across utterances generated from identical text.

  • G TRANSFORMER-BASED ATTENTION ALIGNER BASELINE: The baseline uses transformer attention with output positional features as queries and combined input positional features and encoder outputs as keys.Its encoder uses the same dilated convolution stack as the EATS model and applies Layer Normalization.
  • G TRANSFORMER-BASED ATTENTION ALIGNER BASELINE: Figure 6 plots token positions over time for 128 utterances generated from the same text with different latent vectors z.Close-ups show variability at the beginning and end of the sequence.
  • G TRANSFORMER-BASED ATTENTION ALIGNER BASELINE: Figure 7 shows a histogram of lengths for the 128 utterances generated from identical text with different latent vectors z.The histogram complements the token-position plots by summarising predicted sequence-length variation.

H VARIATION IN ALIGNMENT

The aligner uses latent vector z to model variation in token lengths across renditions of the same text. In the reported example, the generated-length variation is subtle but measurable.

  • H VARIATION IN ALIGNMENT: The experiment generated 128 renditions of the same sentence using different latent vectors z to test variation in token lengths.Figures 6 and 7 visualise token positions and utterance-length distributions for these renditions.
  • H VARIATION IN ALIGNMENT: The reported example is the second sentence of the abstract, which describes end-to-end synthesis from character or phoneme sequences to raw speech audio.This identifies the text used for the alignment-variation demonstration.
  • H VARIATION IN ALIGNMENT: Less than 2% variation occurred in the lengths of the generated utterances for the tested sentence.The paper describes this variation as subtle but noticeable.
  • H VARIATION IN ALIGNMENT: The modest variation is consistent with training data consisting of high-quality recordings by professional voice actors.The paper states that only a modest degree of variation is expected in that setting.

I EVALUATION WITH FRÉCHET DEEPSPEECH DISTANCE

The paper finds Fréchet DeepSpeech Distance unreliable for evaluating reasonable-quality models in this setting. It attributes the mismatch with human opinion scores to feature overlap, sample differences, and conditional-alignment requirements.

  • I EVALUATION WITH FRÉCHET DEEPSPEECH DISTANCE: FDSD scores for reasonable-quality models were not aligned with their Mean Opinion Scores in the reported ablations.The paper notes that FDSD could distinguish models that trained from those that did not during early iteration, but not quality reliably thereafter.
  • I EVALUATION WITH FRÉCHET DEEPSPEECH DISTANCE: Spectrogram-based features may explain FDSD’s weaker performance because the models use spectrogram-derived features alongside raw-audio and mel-spectrogram losses.The paper suggests that convincing spectrograms can coexist with lower-quality speech.
  • I EVALUATION WITH FRÉCHET DEEPSPEECH DISTANCE: The MOS and FDSD evaluations used different samples, with MOS testing more challenging and often longer utterances lacking ground-truth audio.FDSD used randomly held-out training samples, whereas MOS used samples for which FDSD could not be computed.
  • I EVALUATION WITH FRÉCHET DEEPSPEECH DISTANCE: Conditional FDSD is not immediately applicable because it requires fixed-length 2-second samples with aligned conditionings.The paper’s setting lacks fixed alignment between ground-truth characters and audio.
  • I EVALUATION WITH FRÉCHET DEEPSPEECH DISTANCE: The authors call for future work on a reliable automatic quantitative quality metric for text-to-speech models in this regime.This recommendation follows the reported disagreement between FDSD and MOS.

J COMPARISON OF TTS METHODS

EATS differs from prior TTS approaches by learning a feed-forward mapping from text or phonemes to audio end-to-end in a single stage. It does so without distillation or separate vocoder training.

  • J COMPARISON OF TTS METHODS: EATS learns a feed-forward text/phoneme-to-audio mapping end-to-end in a single stage.The comparison distinguishes this structure from prior approaches using multiple stages.
  • J COMPARISON OF TTS METHODS: EATS requires neither distillation nor separate vocoder training.The paper identifies the absence of these additional stages as a distinguishing feature.
  • J COMPARISON OF TTS METHODS: ClariNet is also trained in a single stage but uses teacher forcing and remains autoregressive.A separate distillation stage is required to obtain a feed-forward ClariNet model.
Loading 2006.03575v3…