PaperlayerView exact revision on arXiv

Source-linked AI summary

High Fidelity Speech Synthesis with Adversarial Networks

Mikołaj Bińkowski, Jeff Donahue, Sander Dieleman, Aidan Clark, Erich Elsen, Norman Casagrande, Luis C. Cobo, Karen Simonyan

arXiv:1909.11646v2cs.SDcs.LGeess.AS

TL;DR

Audio GANs had received limited attention while autoregressive models remained state of the art for speech generation. This paper introduces GAN-TTS, combining a conditional feed-forward raw-waveform generator with random-window discriminators and DeepSpeech-based metrics. GAN-TTS produces high-fidelity speech with MOS comparable to WaveNet while enabling efficient generation through its feed-forward design.

  • Problem

    GANs had shown limited success in audio generation, while autoregressive speech models remained difficult to parallelise over time.

  • Method

    GAN-TTS uses a conditional convolutional feed-forward generator for raw audio, an ensemble of random-window discriminators, and DeepSpeech-based speech-generation metrics.

  • Results

    A best-performing model achieves a MOS of 4.2 versus WaveNet's MOS of 4.4, while the proposed metrics rank models in line with MOS.

  • Takeaways & Limitations

    GAN-TTS demonstrates that adversarially trained feed-forward generators can produce high-fidelity speech and support efficient audio generation.

Abstract

from arXiv · show

Generative adversarial networks have seen rapid development in recent years and have led to remarkable improvements in generative modelling of images. However, their application in the audio domain has received limited attention, and autoregressive models, such as WaveNet, remain the state of the art in generative modelling of audio signals such as human speech. To address this paucity, we introduce GAN-TTS, a Generative Adversarial Network for Text-to-Speech. Our architecture is composed of a conditional feed-forward generator producing raw speech audio, and an ensemble of discriminators which operate on random windows of different sizes. The discriminators analyse the audio both in terms of general realism, as well as how well the audio corresponds to the utterance that should be pronounced. To measure the performance of GAN-TTS, we employ both subjective human evaluation (MOS - Mean Opinion Score), as well as novel quantitative metrics (Fréchet DeepSpeech Distance and Kernel DeepSpeech Distance), which we find to be well correlated with MOS. We show that GAN-TTS is capable of generating high-fidelity speech with naturalness comparable to the state-of-the-art models, and unlike autoregressive models, it is highly parallelisable thanks to an efficient feed-forward generator. Listen to GAN-TTS reading this abstract at https://storage.googleapis.com/deepmind-media/research/abstract.wav.

1 INTRODUCTION

GAN-TTS applies adversarial training to raw-waveform text-to-speech, addressing limited GAN success in audio while targeting efficient parallel generation. It combines a feed-forward generator, complementary discriminators, speech metrics, and evaluation against WaveNet.

  • Autoregressive waveform models such as WaveNet are difficult to parallelise over time because they predict each audio step sequentially.This sequential computation is described as expensive and often impractical.
  • GANs had achieved high-fidelity image generation, but their application to audio generation had seen relatively limited success.
  • GAN-TTS uses a convolutional feed-forward generator with discriminators operating on multi-frequency random windows.The discriminators jointly assess generated-audio realism and, for conditioned variants, correspondence to the input utterance.
  • The paper proposes speech-generation metrics based on replacing the image Inception network in FID and KID with DeepSpeech.The resulting family includes unconditional and conditional Fréchet and Kernel DeepSpeech distances.
  • A best-performing model achieves a MOS of 4.2, compared with WaveNet's MOS of 4.4.The authors describe this result as comparable to the state of the art and as establishing GANs as a viable option for efficient TTS.

2 RELATED WORK

Related work contrasts likelihood-based and waveform or spectrogram-based audio generation with GAN approaches. Before GAN-TTS, successful audio GAN demonstrations were largely limited to simpler speech-command, instrument-note, or intermediate spectrogram settings.

  • Likelihood-based audio models include autoregressive models that factorise the joint distribution and invertible feed-forward models that model joint density directly.
  • Waveform-domain models directly generate amplitude over time, whereas many discriminative audio models use spectrograms that encode perceptual biases and discard phase information.
  • Tacotron and MelNet reconstruct missing phase with Griffin-Lim, while Deep Voice and Tacotron 2 generate spectral representations before separate autoregressive waveform synthesis.
  • GANs use adversarially trained generators and discriminators and had achieved state-of-the-art results in image generation.
  • Prior successful audio GAN applications targeted spoken digits, single instrument notes, or magnitude spectrograms rather than large-scale text-to-speech raw audio.WaveGAN generated raw audio for spoken commands, GANSynth generated invertible note spectrograms, and an adversarial vocoder generated magnitude spectrograms from mel-spectrograms.
  • The authors state that GANs had not yet been applied at large scale to non-visual domains, noting that two seconds of 24kHz audio has dimensionality 48000.
  • Multiple discriminators had been used for different image resolutions and for complementary video representations.

3 GAN-TTS

GAN-TTS uses a feed-forward raw-waveform generator and an ensemble of conditional and unconditional random-window discriminators. The architecture combines multi-scale audio evaluation with alignment to linguistic and pitch conditioning.

  • Generator: The generator maps 200Hz linguistic and pitch features to 24kHz raw waveform audio through seven residual blocks with dilated convolutions.The generator uses dilation factors 1, 2, 4, and 8 within its kernel-size-3 convolutions, with noise-conditioned batch normalization.
  • Random-window discriminators: Random Window Discriminators sample waveform fragments at multiple window sizes instead of evaluating each full generated sample.The best-performing model uses window sizes of 240, 480, 960, 1920, and 3600 samples.
  • Random-window discriminators: Different-size random windows augment the data and reduce discriminator computational complexity by reshaping inputs to a constant temporal dimension.Inputs are reshaped from [ωk, 1] to [ω, k] with ω = 240, keeping RWD architectures and approximate complexity similar.
  • Conditional and unconditional evaluation: Conditional discriminators assess correspondence between generated audio and linguistic or pitch features, whereas unconditional discriminators assess realism without conditioning.Conditional windows are aligned every 5ms at 200Hz; unconditional windows are sampled freely at 24kHz.
  • Conditional and unconditional evaluation: The final discriminator ensemble contains 10 RWDs, combining five unconditional and five conditional discriminators across the selected window sizes.The architecture also supports alternative RWD combinations and a full deterministic discriminator for ablation experiments.

4 EVALUATION

GAN-TTS is evaluated through human naturalness judgments and speech-specific distributional metrics. The evaluation compares subjective scores with WaveNet and Parallel WaveNet and adapts Fréchet and Kernel distances using DeepSpeech representations.

  • Human evaluation: Human evaluators rated the naturalness of 1000 sentences on a 1–5 Likert scale, alongside reported WaveNet and Parallel WaveNet scores.The evaluation used subjective Mean Opinion Scores (MOS).
  • Speech distances: The proposed speech metrics include unconditional and conditional Fréchet DeepSpeech Distance and Kernel DeepSpeech Distance.The conditional variants account for linguistic conditioning, while the unconditional variants do not.
  • Speech distances: The metrics extract speech features from the DeepSpeech2 recognition model and compare generated and reference distributions in that representation space.Fréchet distance and Maximum Mean Discrepancy underlie the two metric families.
  • Speech distances: Both metric families are estimated from 10,000 generated and reference samples drawn independently under matching or independent conditioning schemes.Matching linguistic features are used for conditional estimates; independent linguistic features are used for unconditional estimates.
  • Speech distances: Kernel distances complement Fréchet distances because their unbiased MMD estimator avoids the bias issue associated with FID-style estimation.The unbiased estimator also yields zero for real data, enabling comparison in the conditional case.

5 EXPERIMENTS

The experiments compare GAN-TTS with WaveNet and ablated discriminator designs using quantitative metrics and human ratings. The best model is comparable to strong baselines, while multiple random-window discriminators and varied window sizes improve performance and the proposed metrics correlate with MOS.

  • 5.1 RESULTS: The experiments compare GAN-TTS with WaveNet, Parallel WaveNet, and several ablations using quantitative metrics and mean opinion scores.Table 1 reports the proposed model, prior-work benchmarks, and alternative GAN-TTS variants.
  • 5.1 RESULTS: The best model achieves worse yet comparable scores to WaveNet and Parallel WaveNet, although dataset differences prevent direct comparison.The baselines used 65 hours of training data, slightly more than GAN-TTS.
  • 5.1 RESULTS: Multiple random-window discriminators outperform a single conditional discriminator, while unconditional discriminators further improve results.The deterministic full discriminator achieves the worst scores among the compared variants.
  • 5.1 RESULTS: Among 10-discriminator models, combining different window sizes is beneficial, and the downsampling model with base window size 240 performs best.A simple ensemble of 10 fixed-size windows performs significantly worse than combinations using varied discriminator sizes.
  • 5.1 RESULTS: Human evaluation scores show a noticeable correlation with the proposed quantitative metrics.This supports using the metrics to evaluate neural audio synthesis models.
  • 5.2 DISCUSSION: The generator uses 0.64 MFLOP/sample and a 590ms receptive field, compared with 1.97 MFLOP/sample and 320ms for Parallel WaveNet.The discriminator ensemble operates on 10ms–150ms windows, making training faster than with a full discriminator.
  • 5.2 DISCUSSION: Training remained stable for up to 1 million steps, with gradual subjective improvement and decreasing metric values without observed model collapse.The paper contrasts this with model collapses often reported in GAN literature.

6 CONCLUSION

The paper concludes that GAN-TTS enables raw-audio text-to-speech generation with a feed-forward convolutional generator and an ensemble of conditional and unconditional random-window discriminators. Its evaluations support high-fidelity synthesis, efficient generation, useful DeepSpeech-based metrics, and the feasibility of GANs for text-to-speech.

  • 6 CONCLUSION: GAN-TTS generates raw speech audio with an adversarially trained feed-forward convolutional network.The feed-forward design allows efficient audio generation.
  • 6 CONCLUSION: Conditional and unconditional Random Window Discriminators operating at different window sizes assess text correspondence and speech realism, respectively.The ablation study found each component instrumental to good performance.
  • 6 CONCLUSION: Conditional and Kernel DeepSpeech Distance metrics rank speech-generation models in line with human Mean Opinion Scores.The metrics are based on the publicly available DeepSpeech recognition model.
  • 6 CONCLUSION: Quantitative and subjective evaluations showcase the feasibility of text-to-speech generation with GANs.

A.1 MASKING CONVOLUTIONS TO GENERATE LONGER SAMPLES

The fully convolutional generator can theoretically produce samples of arbitrary length, but batching requires variable-length inputs to be zero-padded into fixed-size tensors.

  • A.1 MASKING CONVOLUTIONS TO GENERATE LONGER SAMPLES: The fully convolutional generator is theoretically capable of generating samples of arbitrary length.In practice, deep-learning frameworks commonly process fixed-size batches.
  • A.1 MASKING CONVOLUTIONS TO GENERATE LONGER SAMPLES: Variable-length inputs are zero-padded to fit fixed-size batch tensors.Convolutional layers also commonly pad inputs to obtain desired output dimensions.

A.2 ARCHITECTURE DETAILS

The appendix specifies the generator and discriminator architectures, including temporal and channel dimensions, discriminator downsampling factors, and conditional feature-waveform integration. DeepSpeech-based metrics extract and average high-level features from overlapping raw-audio windows.

  • A.2 ARCHITECTURE DETAILS: Table 2 describes GAN-TTS’s generator architecture using temporal dimension t, channel count ch, and output dimensions for each layer.The generator has 30 layers, most belonging to dilated residual blocks.
  • A.2 ARCHITECTURE DETAILS: Table 3 lists discriminator downsample factors for different initial stride values k, including non-downsampling residual blocks in the block count.
  • A.2 ARCHITECTURE DETAILS: Conditional discriminators add waveform and linguistic-feature representations after downsampling the waveform temporal dimension by a factor of 120.Downsampling uses an initial reshape followed by residual blocks whose factors are prime divisors of 120/k.
  • A.2 ARCHITECTURE DETAILS: The evaluation function extracts 1600-dimensional features from each 20ms raw-audio window, sampled every 10ms, then averages them over time.Features come from the pre-trained DeepSpeech2 model.

B.2 METRICS IN DISTRIBUTION SPACE

The paper adapts Fréchet distance and MMD to speech by comparing DeepSpeech2 representations of real and generated audio, including conditional text-to-speech distributions. It also uses µ-law-transformed audio, which early experiments found improved performance.

  • KID uses an unbiased estimator, whereas FID estimates remain positively biased even when real samples come from the same distribution.The same estimator properties apply to the proposed DeepSpeech metrics.
  • Conditional cFDSD cannot be estimated on real data because two independent real samples with identical conditioning cannot be sampled.For cKDSD, the corresponding real-data estimate would be close to zero if such samples were available.
  • Conditional distances compare generated and real distributions using the same linguistic conditioning, p(xG|c) and p(xreal|c).Unconditional distances instead compare p(xG) and p(xreal).
  • 16-bit µ-law-transformed audio performed better in early experiments than non-transformed waveforms.The transform was used without quantisation because the model operates in the continuous domain.

D TRAINING DETAILS

Training used adversarial optimization with stabilization techniques, distributed TPU computation, and standing BatchNorm statistics for variable-length inference. The reported cFDSD learning curve decreased stably and gradually.

  • Training used one discriminator update per generator update, with learning rates of 10^-4 and 5 × 10^-5 respectively.The models used hinge loss, batch size 1024, and Adam with β1 = 0 and β2 = 0.999.
  • Spectral normalization, orthogonal initialization, orthogonal regularization, exponential moving averaging, and cross-replica BatchNorm stabilized the models.Generator weights used exponential moving averaging with decay 0.9999 for sampling.
  • Standing BatchNorm statistics made generator layers independent of inference-sample characteristics, supporting audio generation of unspecified length.Statistics were accumulated before sampling rather than taken from variable-size inference batches.
  • Models were trained for 1 million generator and discriminator updates on Cloud TPU v3 Pods using 128-way data parallelism.Training usually took up to 48 hours.
  • cFDSD decreased stably and gradually during training.The model also showed gradual improvement in subjective sample quality and no observed collapse during training.
Loading 1909.11646v2…