Source-linked AI summary
Parallel WaveGAN: A fast waveform generation model based on generative adversarial networks with multi-resolution spectrogram
Ryuichi Yamamoto, Eunwoo Song, Jae-Min Kim
TL;DR
Slow autoregressive generation and cumbersome teacher-student density distillation limit fast neural vocoding. Parallel WaveGAN trains a compact non-autoregressive WaveNet with adversarial and multi-resolution STFT losses, achieving fast 24 kHz synthesis and competitive perceptual quality.
Problem
Autoregressive WaveNet is slow for real-time use, while teacher-student methods require a trained teacher and complicated density-distillation optimization.
Method
Parallel WaveGAN trains a non-autoregressive WaveNet by jointly optimizing waveform-domain adversarial loss and multi-resolution STFT loss without density distillation.
Results
4.16 MOS was achieved in Transformer-based TTS, while 24 kHz speech generation reached 28.68 times real-time with 1.44 M parameters.
Takeaways & Limitations
The method provides distillation-free waveform generation that is fast, compact, and competitive with conventional distillation-based approaches.
Takeaways & Limitations
The model details use band-limited features to alleviate over-smoothing caused by TTS acoustic models.
Abstract
from arXiv · showhide
We propose Parallel WaveGAN, a distillation-free, fast, and small-footprint waveform generation method using a generative adversarial network. In the proposed method, a non-autoregressive WaveNet is trained by jointly optimizing multi-resolution spectrogram and adversarial loss functions, which can effectively capture the time-frequency distribution of the realistic speech waveform. As our method does not require density distillation used in the conventional teacher-student framework, the entire model can be easily trained. Furthermore, our model is able to generate high-fidelity speech even with its compact architecture. In particular, the proposed Parallel WaveGAN has only 1.44 M parameters and can generate 24 kHz speech waveform 28.68 times faster than real-time on a single GPU environment. Perceptual listening test results verify that our proposed method achieves 4.16 mean opinion score within a Transformer-based text-to-speech framework, which is comparative to the best distillation-based Parallel WaveNet system.
1. INTRODUCTION
Parallel WaveGAN addresses slow autoregressive waveform generation and cumbersome teacher-student distillation with a distillation-free GAN-based parallel method. It jointly uses multi-resolution STFT and adversarial losses, reducing training and inference time while maintaining perceptual quality.
- Autoregressive WaveNet provides high-quality speech but is too slow for many real-time applications.
- Teacher-student methods enable faster generation but require a trained teacher and trial-and-error optimization of probability density distillation.
- Parallel WaveGAN trains a non-autoregressive WaveNet directly with jointly optimized multi-resolution STFT and waveform-domain adversarial losses.
- 4.82 times faster training and 1.96 times faster inference were achieved versus ClariNet, generating 24 kHz waveforms at 28.68 times real-time.
- 4.16 MOS was achieved within a Transformer-based TTS framework, competitive with the best distillation-based ClariNet model.
2. RELATED WORK
The related-work discussion distinguishes Parallel WaveGAN from prior GAN and teacher-student approaches by removing density distillation and directly estimating speech waveforms. Joint adversarial and multi-resolution STFT optimization is used to model speech dynamics without intermediate linear-prediction parameters.
- Prior GAN-based Parallel WaveNet work improved perceptual quality but retained a complicated density-distillation training stage.
- The proposed method aims to minimize the training effort of the conventional two-stage teacher-student pipeline by eliminating distillation.
- Unlike GELP, Parallel WaveGAN directly estimates speech waveforms, avoiding dependence on linear-prediction parameters that may contain acoustic-model errors.
3. METHOD
The method combines a non-autoregressive WaveNet generator with adversarial training and a multi-resolution STFT auxiliary loss to model realistic speech waveforms.
- Parallel waveform generation: The generator transforms Gaussian noise into waveforms in parallel using non-causal convolutions and remains non-autoregressive during training and inference.It is conditioned on an auxiliary feature such as a mel-spectrogram.
- Adversarial training: The adversarial objective trains the generator to make generated samples appear real to a discriminator, while the discriminator separates generated and target waveforms.The target waveform is denoted by x, and its distribution by pdata.
- Multi-resolution STFT auxiliary loss: The multi-resolution STFT auxiliary loss sums STFT losses computed with different FFT sizes, window sizes, and frame shifts.This exposes the generator to multiple time-frequency representations rather than a single fixed STFT configuration.
- Multi-resolution STFT auxiliary loss: Combining multiple STFT resolutions helps the generator learn speech time-frequency characteristics and prevents overfitting to one fixed STFT representation.STFT analysis trades temporal resolution against frequency resolution as the window size changes.
- Joint objective: The final generator objective is a linear combination of the multi-resolution STFT loss and waveform-domain adversarial loss.The hyperparameter λadv balances the two loss terms.
4. EXPERIMENTS
Experiments evaluate Parallel WaveGAN’s waveform quality, inference speed, training time, and use within Transformer-based TTS. The model combines multi-resolution STFT and adversarial objectives to achieve competitive perceptual quality with simpler training.
- Waveform generation evaluation: Multi-resolution STFT loss improved perceptual quality over single-resolution STFT loss and autoregressive WaveNet without STFT loss.The evaluation attributes this improvement to better capture of speech time-frequency characteristics and full-band frequency information.
- Evaluation: 24 kHz speech generation was evaluated using inference speed and MOS tests on a single NVIDIA Tesla V100 GPU.MOS tests used 18 native Japanese speakers and 20 randomly selected utterances.
- Efficiency: Parallel WaveGAN generated speech 1.96 times faster than ClariNet and required 2.8 training days.Its training time was 2.64 times faster than autoregressive WaveNet and 4.82 times faster than ClariNet because density distillation was unnecessary.
- Text-to-speech evaluation: The Transformer-based TTS evaluation used phoneme and accent sequences to generate mel-spectrograms, which the vocoders converted into time-domain speech.The Transformer used six-layer encoder and decoder stacks with eight-head multi-head attention.
- Text-to-speech evaluation: 4.16 MOS was achieved by Parallel WaveGAN with Transformer TTS, comparable to the best distillation-based Parallel WaveNet system.The evaluation used acoustic features generated by the Transformer TTS model as vocoder inputs.
5. CONCLUSION
Parallel WaveGAN is a distillation-free GAN waveform generator that jointly uses adversarial and multi-resolution STFT losses. It achieved competitive TTS quality while generating speech rapidly with a compact model.
- Conclusion: Parallel WaveGAN uses a non-autoregressive model trained with waveform-domain adversarial loss and multi-resolution STFT loss.The method avoids complicated probability density distillation.
- Conclusion: 4.16 MOS was achieved within a Transformer-based TTS framework, competitive with conventional distillation-based approaches.The model generated 24 kHz speech 28.68 times faster than real-time with 1.44 M parameters.
- Conclusion: Future work includes improving the multi-resolution STFT auxiliary loss and evaluating performance on varied, including expressive, corpora.The proposed phase-related loss is one example of a possible improvement.