End-to-End Adversarial Text-to-Speech

• EATS uses adversarial training to map text or temporally unaligned phonemes directly to raw waveforms, eliminating typical intermediate bottlenecks.Traditional staged pipelines can require stage-specific supervision and sequential training.Source: §1 · p. 1
• A fully differentiable, efficient feed-forward aligner predicts each input token's duration and creates audio-aligned features, which a decoder upsamples into 24 kHz waveforms.Source: §1 · pp. 1–2
• Flexible dynamic time warping-based prediction losses enforce alignment with input conditioning while allowing timing variability in human speech.Source: §1 · p. 2
• 4.083 mean opinion score: EATS approaches state-of-the-art TTS systems trained with richer supervisory signals.Source: §1 · p. 2

2 METHOD

• The end-to-end generator maps characters or phonemes to 24-kHz raw audio through a 200-Hz aligner and upsampling decoder, generating speech at 200× realtime on one V100 GPU.The speaker-conditional decoder uses latent and speaker embeddings; training combines adversarial feedback with a hinge-loss discriminator objective.Source: §2 · p. 2

2.1 ALIGNER

• The aligner predicts nonnegative token lengths, cumulatively derives monotonic end positions, and softly interpolates token representations into 200-Hz aligned features.Its dilated convolutions let tokens such as punctuation affect utterance-wide prosody and support generalisation across sequence lengths.Source: §2.1 · pp. 2–3

2.2 WINDOWED GENERATOR TRAINING

• Training uses randomly offset 2-second windows instead of padding variable-length utterances; evaluation generates the full utterance with the fully convolutional model.The aligner computes features only for the sampled window but predicts token lengths across the entire input sequence.Source: §2.2 · p. 3

2.3 ADVERSARIAL DISCRIMINATORS

• Five speaker-conditioned random-window discriminators inspect short audio fragments at different resolutions, alongside a speaker-conditioned spectrogram discriminator on the full training window.The random-window sizes are 240, 480, 960, 1920, and 3600 samples; the spectrogram discriminator uses a BigGAN-deep architecture.Source: §2.3 · pp. 3–4

2.4 SPECTROGRAM PREDICTION LOSS

• Adversarial feedback alone cannot bootstrap alignment, so an L1 loss between generated and ground-truth log-mel spectrograms guides training while the generator produces raw waveforms.The spectrogram-domain loss is less sensitive to perceptually insignificant phase differences; small waveform jitter of up to ±60 samples helped reduce artifacts.Source: §2.4 · p. 4

2.5 DYNAMIC TIME WARPING

• Soft dynamic time warping aggregates costs across feasible paths, allowing generated and target spectrograms to differ in timing while still assuming aligned training-window endpoints.A warp penalty of 1.0 discourages large deviations from identity alignment; the soft-minimum temperature is 0.01.Source: §2.5 · pp. 4–5

2.6 ALIGNER LENGTH LOSS

• A length loss encourages the sum of predicted token lengths to match the ground-truth utterance length, since individual ground-truth token durations are unavailable.The loss scale factor is λlength = 0.1.Source: §2.6 · p. 5

2.7 TEXT PRE-PROCESSING

• Phoneme input significantly improves sample quality over character input; preprocessing uses partial normalisation and phonemisation and pads sequences with silence tokens.The silence tokens let the aligner account for silence at utterance beginnings and ends.Source: §2.7 · p. 5

3 RELATED WORK

• Autoregressive waveform models have sequential dependencies that hinder real-time use, while flow-based models offer fast feed-forward inference but condition on spectrograms or linguistic features.Flow-based models may use maximum-likelihood training (Prenger et al., 2019; Kim et al., 2019; Ping et al., 2019b) or distillation from autoregressive models (van den Oord et al., 2018; Ping et al., 2019a).Source: §3 · p. 6
• Adversarial TTS prioritizes realism over diversity; MelGAN (Kumar et al., 2019) and Parallel WaveGAN (Yamamoto et al., 2020) are mel-conditioned vocoders, while GAN-TTS (Bi´nkowski et al., 2020) uses aligned linguistic features.The paper identifies GAN-TTS as its most direct predecessor and builds on it.Source: §3 · p. 6
• Text-to-spectrogram systems require vocoding; they include autoregressive models, distilled non-autoregressive models such as ParaNet (Peng et al., 2019) and FastSpeech (Ren et al., 2019), and feed-forward flow models such as Flow-TTS and Glow-TTS (Kim et al., 2020).Because predicted spectrograms are imperfect, most such systems need a custom vocoder trained on generated spectrograms.Source: §3 · p. 6
• The proposed model generates waveforms from character or phoneme sequences in one feed-forward stage, learning alignment without auxiliary supervision or teacher forcing.Char2wav is also finetuned end-to-end, but requires pretraining with vocoder features for intermediate supervision.Source: §3 · p. 6
• Soft dynamic time warping relaxes spectrogram-loss alignment constraints, whereas Monotonic Alignment Search (Kim et al., 2020) aligns text with a latent representation; the model enforces monotonicity through non-recurrent interpolation.Its L1 spectrogram loss is comparatively simple, unlike literature losses that combine magnitude, log-magnitude, and phase terms, often across resolutions.Source: §3 · p. 7

4 EVALUATION

• Speech quality is evaluated primarily by mean naturalness ratings from 1,000 held-out conditioning sequences, with MOS on a 1–5 scale.The section also reports FDSD in Appendix I.Source: §4 · p. 7

4.1 MULTI-SPEAKER DATASET

• Training uses 260.49 hours of 24 kHz recordings from 69 North American English voices; main MOS results use the most prolific speaker, with top-four-speaker results also reported.The dataset contains full-sentence clips from under 1 to 20 seconds, and training samples two-second windows.Source: §4.1 · p. 7

4.2 RESULTS

• EATS scores MOS 4.083 versus prior MOS 4.2–4.4+; dataset differences limit direct comparison, though EATS uses substantially less supervision than aligned-feature models.The cited prior results are from Bi´nkowski et al. (2020), van den Oord et al. (2016; 2018).Source: §4.2 · p. 7, Table 1 · p. 8
• Removing RWDs, the mel-spectrogram discriminator, or all discriminators substantially worsens MOS; No RWDs scores 2.526, while removing length or prediction losses prevents training.Without RWDs, high-frequency audio degrades; without the mel-spectrogram discriminator, artifacts and distortion occur, and without discriminators, speech sounds robotic and distorted.Source: §4.2 · p. 7
• No DTW, raw-character inputs, and attention alignment yield MOS 3.559, 3.423, and 3.551, respectively; attention alignment generalizes poorly to long utterances.Removing DTW causes warbling and unnatural phoneme lengths, while raw-character input can cause mispronunciations and unusual stress patterns.Source: §4.2 · pp. 7–8
• Single-speaker training scores MOS 3.829, while the multi-speaker model benefits from more training data; MOS generally rises with speaker data, but imperfectly.The same multi-speaker EATS model is evaluated on the four most prolific speakers, and the aligner uses latent vector z to vary predicted token lengths.Source: §4.2 · p. 8, Table 2 · p. 8

5 DISCUSSION

• The adversarial TTS model learns from normalized text or phonemes paired with speech, generalizes to unseen text, and approaches multi-stage or additionally supervised systems in human-rated naturalness.It is efficient in training and inference and avoids autoregressive sampling and teacher forcing.Source: §5 · p. 9
• The method still trails state-of-the-art speech fidelity, but end-to-end learning could benefit from more data and optimize intermediate representations beyond typical pipeline bottlenecks.The 69-speaker main result used roughly four times the data of the Single Speaker ablation and produced more natural speech in that ablation's single voice.Source: §5 · p. 9
• The current method relies on a separate, fixed system for text normalization and phonemization, whereas fully end-to-end TTS could use raw text.The authors believe a data-driven approach could ultimately succeed in this setting given sufficient training data and model capacity.Source: §5 · p. 9

A HYPERPARAMETERS AND OTHER DETAILS

• Training ran for 5 × 10^5 steps, each comprising one discriminator update followed by one generator update with independently sampled minibatches of 1,024.Source: A HYPERPARAMETERS AND OTHER DETAILS · p. 14
• Both updates used Adam with β1 = 0, β2 = 0.999, and learning rate 10^-3, cosine-decayed to zero at step 500K.Source: A HYPERPARAMETERS AND OTHER DETAILS · p. 14
• Spectral normalization was applied to the decoder and discriminators, but not the aligner; parameters were initialized orthogonally, with generator off-diagonal orthogonal regularization weighted 10^-4.Source: A HYPERPARAMETERS AND OTHER DETAILS · p. 14
• Minibatches were split across 64 or 128 TPU v3 cores, enabling single-model training in up to 58 hours, with batch statistics aggregated across devices.Source: A HYPERPARAMETERS AND OTHER DETAILS · p. 14
• Sampling used 200 forward passes to accumulate standing batch-normalization statistics and generator-weight EMA with decay 0.9999; training consistently converged.The implementation used TensorFlow v1, Sonnet, and TF-Replicator for data-parallel TPU training.Source: A HYPERPARAMETERS AND OTHER DETAILS · p. 14
• 206.8× realtime on GPU, 157.2× per TPU chip, and 8.52× on CPU are the reported EATS inference factors.The GPU benchmark generated 2,400 seconds of speech in an average (median) 11.60 seconds; the TPU rate was 78.6× per core.Source: A HYPERPARAMETERS AND OTHER DETAILS · p. 14

B ALIGNER PSEUDOCODE

• Figure 3 presents pseudocode for the EATS aligner described in Section 2.1.Source: B ALIGNER PSEUDOCODE · p. 15

C SPECTROGRAM DISCRIMINATOR ARCHITECTURE

• Its input is a 47 × 80 × 1 image formed from the mel-spectrogram of a 48,000-sample waveform, equivalent to 2 seconds at 24 kHz.Source: C SPECTROGRAM DISCRIMINATOR ARCHITECTURE · p. 15
• The discriminator follows the BigGAN-deep 128×128 image architecture, omitting its first two ResBlocks and self-attention block to remove one 2 × 2 downsampling step.The omitted blocks are rows 2–4; the input convolution and subsequent rows are retained.Source: C SPECTROGRAM DISCRIMINATOR ARCHITECTURE · p. 15
• The architecture uses a channel-width multiplier of ch = 64.Source: C SPECTROGRAM DISCRIMINATOR ARCHITECTURE · p. 15

D MEL-SPECTROGRAM COMPUTATION

• Mel-spectrograms for prediction losses use jittered real spectrograms and unjittered generated spectrograms, while discriminator inputs use no jitter for either.The computation function supplies both the spectrogram discriminator and the spectrogram prediction loss.Source: D MEL-SPECTROGRAM COMPUTATION · p. 17

E DYNAMIC TIME WARPING PSEUDOCODE

• Figure 5 gives the pseudocode for the soft dynamic time warping procedure used in the spectrogram prediction loss.The procedure is used for L′′pred.Source: E DYNAMIC TIME WARPING PSEUDOCODE · p. 18, Figure 5 · p. 18
• The DTW implementation has quadratic complexity, and although Itakura or Sakoe-Chiba bands could improve efficiency, enabling DTW did not meaningfully affect training time.The authors therefore found this optimization unnecessary in practice.Source: E DYNAMIC TIME WARPING PSEUDOCODE · p. 18

F TEXT PREPROCESSING

• For all results except the No Phonemes ablation, input text is partially normalized and phonemized with phonemizer 2.2 and its espeak backend, using espeak-ng 1.50 to produce IPA sequences.The No Phonemes ablation instead uses character sequences directly.Source: F TEXT PREPROCESSING · p. 19
• Phonemizer is configured to include primary and secondary stress marks and to remove spurious whitespace.Source: F TEXT PREPROCESSING · p. 19
• Punctuation is preserved because it can meaningfully affect prosody.Source: F TEXT PREPROCESSING · p. 19
• Rare phonemizer symbols, usually found in non-English words, are replaced with more frequent symbols, yielding a set of 51 distinct symbols.Table 4 lists the symbols that are replaced or removed.Source: F TEXT PREPROCESSING · p. 19, Table 4 · p. 19

G TRANSFORMER-BASED ATTENTION ALIGNER BASELINE

• The baseline uses transformer attention (Vaswani et al., 2017) to align encoded text features to audio positions, then applies Batch Normalisation, ReLU, and decoder upsampling.It uses sinusoidal positional features, four heads with key and value dimensions of 64 per head, and encoder outputs from the EATS dilated convolution stack.Source: G TRANSFORMER-BASED ATTENTION ALIGNER… · p. 19
• Figure 6 shows token positions over time for 128 utterances generated from the same text with different latent vectors, with start- and end-sequence close-ups showing predicted-length variability.The attention uses output positional features as queries and summed input positional features and encoder outputs as keys.Source: G TRANSFORMER-BASED ATTENTION ALIGNER… · p. 19, Figure 6 · p. 20
• Figure 7 presents a histogram of lengths for 128 utterances generated from the same text with different latent vectors.Source: Figure 7 · p. 20

H VARIATION IN ALIGNMENT

• The experiment generated 128 renditions of the abstract’s second sentence to test whether the aligner uses latent vector z to account for token-length variation.Source: H VARIATION IN ALIGNMENT · p. 20
• Variation in generated utterance length was less than 2% but noticeable for the tested sentence.Figures 6 and 7 show token positions over time and a histogram of generated utterance lengths.Source: H VARIATION IN ALIGNMENT · p. 20
• Only modest variation was expected because the training data contains high-quality recordings of human speech performed by professional voice actors.Source: H VARIATION IN ALIGNMENT · p. 20

I EVALUATION WITH FRÉCHET DEEPSPEECH DISTANCE

• FDSD distinguished models that trained from those that did not early in iteration, but scores for reasonable-quality models did not align with MOS.The authors suggest that spectrogram-derived features may reward convincing spectrograms despite lower-quality audio, and that FDSD and MOS used different, non-equivalent samples and sample sizes.Source: I EVALUATION WITH FRÉCHET DEEPSPEECH… · p. 21
• ≈3.5 MOS for No MelSpecD versus ≈2.5 MOS for No RWDs, although FDSD ranked these ablations oppositely.No RWDs is optimized only against spectrogram-based losses.Source: I EVALUATION WITH FRÉCHET DEEPSPEECH… · p. 21
• Conditional FDSD is not directly applicable because it requires fixed two-second samples with aligned conditioning, unlike this setting’s unaligned characters and audio.Source: I EVALUATION WITH FRÉCHET DEEPSPEECH… · p. 21
• The authors call for future work on a reliable automatic quality metric for text-to-speech models in this regime.Source: I EVALUATION WITH FRÉCHET DEEPSPEECH… · p. 21

J COMPARISON OF TTS METHODS

• EATS learns a feed-forward text/phoneme-to-audio mapping end-to-end in a single stage.It does not require distillation or separate vocoder training.Source: J COMPARISON OF TTS METHODS · p. 23
• ClariNet is also trained in a single stage, but uses teacher forcing and is therefore autoregressive.Source: J COMPARISON OF TTS METHODS · p. 23
• Obtaining a feed-forward ClariNet model requires a separate distillation stage.Source: J COMPARISON OF TTS METHODS · p. 23