Source-linked AI summary
Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speech
Jaehyeon Kim, Jungil Kong, Juhee Son
TL;DR
Existing parallel end-to-end TTS systems have not matched the quality of two-stage systems. VITS combines conditional variational inference, normalizing flows, adversarial training, and stochastic duration prediction; it outperforms two-stage TTS systems and approaches human quality, while text preprocessing remains unresolved.
Problem
Existing parallel end-to-end TTS systems provide simplified training and sampling but have synthesis quality that lags behind two-stage systems.
Method
VITS uses a conditional VAE with normalizing flows, waveform-domain adversarial training, and a stochastic duration predictor to generate parallel speech directly from text.
Results
VITS outperforms two-stage TTS systems and achieves close to human quality in the reported experiments.
Takeaways & Limitations
VITS produces natural-sounding speech waveforms directly from text without predefined intermediate speech representations and supports diverse speech rhythms.
Takeaways & Limitations
The system still requires text preprocessing, and the authors identify self-supervised language representations as a possible direction for removing it.
Abstract
from arXiv · showhide
Several recent end-to-end text-to-speech (TTS) models enabling single-stage training and parallel sampling have been proposed, but their sample quality does not match that of two-stage TTS systems. In this work, we present a parallel end-to-end TTS method that generates more natural sounding audio than current two-stage models. Our method adopts variational inference augmented with normalizing flows and an adversarial training process, which improves the expressive power of generative modeling. We also propose a stochastic duration predictor to synthesize speech with diverse rhythms from input text. With the uncertainty modeling over latent variables and the stochastic duration predictor, our method expresses the natural one-to-many relationship in which a text input can be spoken in multiple ways with different pitches and rhythms. A subjective human evaluation (mean opinion score, or MOS) on the LJ Speech, a single speaker dataset, shows that our method outperforms the best publicly available TTS systems and achieves a MOS comparable to ground truth.
1. Introduction
End-to-end parallel TTS simplifies training and sampling but has lagged two-stage systems in synthesis quality. VITS addresses this gap with variational, flow-based, adversarial modeling and stochastic duration prediction.
- Motivation: Two-stage TTS systems require sequential training or fine-tuning and predefined intermediate features, while recent end-to-end systems still lag in synthesis quality.The cited passage identifies these as limitations of existing pipelines.
- Proposed approach: VITS connects TTS modules through latent variables for efficient end-to-end learning and applies normalizing flows and waveform-domain adversarial training.These design choices target the expressive power and quality of waveform generation.
- Proposed approach: The stochastic duration predictor models diverse rhythms, while latent-variable uncertainty represents multiple spoken realizations of the same text.The method targets variation in pitch and duration that text alone cannot specify.
- Results: VITS obtains more natural speech and higher sampling efficiency than Glow-TTS with HiFi-GAN, described as the best publicly available TTS system.The comparison concerns both perceived naturalness and sampling efficiency.
2. Method
VITS is a conditional VAE that reconstructs speech through latent variables, enriches its prior with normalizing flows, and trains with adversarial and feature-matching losses. Monotonic alignment and stochastic duration modeling support text-to-speech generation with parallel inference and variable rhythms.
- Conditional VAE: VITS maximizes a variational lower bound using reconstruction loss and KL divergence between the approximate posterior and conditional prior.The latent variable z is sampled from qφ(z|x), while the prior is conditioned on c.
- Reconstruction loss: The reconstruction loss compares target and predicted mel-spectrograms after decoding latent variables into waveforms and transforming them through STFT and mel projection.Mel-domain reconstruction is used during training, and windowed generator training processes partial latent sequences.
- Prior and alignment: The prior encoder processes phonemes and alignment information, and a normalizing flow increases the prior distribution’s expressiveness for realistic sampling.The alignment is a hard monotonic attention matrix linking phonemes to latent-variable time steps.
- Prior and alignment: Monotonic Alignment Search uses dynamic programming to find an ELBO-maximizing alignment under monotonic, non-skipping constraints.The method reduces the alignment objective to maximizing the latent-variable log-likelihood and can reuse the original MAS implementation.
- Duration prediction: The stochastic duration predictor models phoneme-duration distributions using variational dequantization and data augmentation, then samples durations from noise through an inverse transformation.These techniques address discrete durations and the scalar dimensionality constraint of invertible flows.
- Adversarial training: Adversarial training adds a discriminator and feature-matching loss, while the posterior encoder and discriminator are used only during training.Feature matching provides a hidden-layer reconstruction signal for the generator.
3. Experiments
Experiments use LJ Speech for comparison with public TTS systems and VCTK for multi-speaker characteristics. The study uses established public implementations, controlled sampling randomness, and windowed training details.
- Datasets: The LJ Speech dataset contains 13,100 single-speaker audio clips totaling approximately 24 hours and is used for comparison with publicly available models.The data are split into training, validation, and held-out portions.
- Training: Training uses AdamW, scheduled learning-rate decay, and windowed generator training to reduce training time and memory usage.The reported optimizer settings include β1 = 0.8, β2 = 0.99, weight decay λ = 0.01, and initial learning rate 2 × 10^-4.
- Baselines: The comparison includes Tacotron 2 and Glow-TTS as first-stage models and HiFi-GAN as a second-stage model, using public implementations and pretrained weights.Fine-tuned HiFi-GAN variants are also included for two-stage comparisons.
- Evaluation controls: Sampling randomness is controlled by fixing model-specific hyperparameters for Tacotron 2, Glow-TTS, and VITS throughout the experiments.VITS uses stochastic-duration noise standard deviation 0.8 and a 0.667 scale factor on prior-distribution standard deviation.
4. Results
Experiments evaluate VITS’s speech quality, ablations, multi-speaker diversity, speech variation, and synthesis efficiency. Results indicate strong naturalness, diverse durations and pitches, and improved sampling efficiency.
- Speech Synthesis Quality: VITS outperforms other TTS systems and achieves a similar MOS to ground truth.The evaluation used crowd-sourced naturalness ratings on a 1–5 scale with 95% confidence intervals.
- Ablation Studies: A 1.52 MOS decrease follows removal of the normalizing flow from the prior encoder.This ablation indicates that prior-distribution flexibility significantly influences synthesis quality.
- Ablation Studies: A -0.19 MOS degradation follows replacing the linear-scale spectrogram with a mel-spectrogram for posterior input.The result indicates that high-resolution information improves VITS synthesis quality.
- Generalization to Multi-Speaker Text-to-Speech: VITS learns and expresses diverse speech characteristics in an end-to-end manner on the VCTK multi-speaker dataset.The multi-speaker evaluation compares VITS with Tacotron 2, Glow-TTS, and HiFi-GAN.
- Speech Variation: VITS produces variable utterance lengths, with a length distribution similar to Tacotron 2, while Glow-TTS generates fixed-length utterances.The comparison uses histograms of 100 generated utterances; multi-speaker samples also imply speaker-dependent phoneme duration.
- Speech Variation: VITS generates diverse pitches and rhythms, including substantially different lengths and pitches across speaker identities.Figure 3 compares pitch tracks for VITS, Tacotron 2, Glow-TTS, and multi-speaker VITS samples.
- Synthesis Speed: VITS improves sampling efficiency and speed because it does not generate predefined intermediate representations.Speed was measured over raw-waveform generation from phoneme sequences for 100 LJ Speech test sentences on one NVIDIA V100 GPU.
5. Related Work
Prior TTS systems trade off end-to-end simplicity, parallel generation, expressive speech, and alignment quality. VITS combines conditional variational modeling, normalizing flows, explicit alignment search, and stochastic duration prediction to address these limitations.
- Two-stage TTS systems require vocoders trained or fine-tuned on earlier-stage outputs and cannot exploit learned hidden representations instead of predefined features.
- Existing single-stage end-to-end models generate raw waveforms directly but generally produce lower audio quality than two-stage systems.
- VITS uses a conditional VAE to synthesize raw waveforms directly from text, estimate alignments with MAS, generate samples in parallel, and outperform publicly available two-stage models.
- Unlike prior VAE applications to first-stage TTS models, VITS applies a VAE to a parallel end-to-end TTS system.
- Normalizing flows augment VITS’s conditional prior distribution to improve its representation power and produce more realistic samples.
- VITS explicitly aligns latent and source sequences with MAS, simplifying its normalizing flows compared with FlowSeq’s implicit attention-based alignment.
- A flow-based stochastic duration predictor models the joint distribution of phoneme durations, enabling diverse speech rhythms during parallel generation.
6. Conclusion
VITS is a parallel end-to-end TTS system that generates natural speech directly from text while modeling diverse rhythms. Experiments report performance above two-stage systems and close to human quality, although text preprocessing remains unresolved.
- VITS learns and generates speech end to end in a parallel TTS system.
- A stochastic duration predictor enables VITS to express diverse speech rhythms.
- VITS synthesizes natural-sounding waveforms directly from text without predefined intermediate speech representations.
- Experimental results show that VITS outperforms two-stage TTS systems and achieves close to human quality.
- Text preprocessing remains a problem, and self-supervised language representations are proposed as a possible direction for removing that step.
Supplementary Material of Conditional Variational Autoencoder with Adversarial Learning for
The supplementary material is identified as belonging to the paper on conditional variational autoencoders with adversarial learning for end-to-end text-to-speech.
- The supplementary material concerns Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speech.
A. Monotonic Alignment Search
The supplementary section presents pseudocode for Monotonic Alignment Search and describes the surrounding VITS architecture components. It identifies the flow, encoder, decoder, and discriminator configurations used in the implementation.
- A. Monotonic Alignment Search: Monotonic Alignment Search is presented as pseudocode in Figure 4.
- A. Monotonic Alignment Search: The MAS function receives a log-likelihood matrix and returns the most likely alignment.
- A. Monotonic Alignment Search: Each matrix entry represents the log-likelihood of a latent variable under a prior mean and variance.
- Implementation details: The architecture reuses Glow-TTS’s transformer encoder and WaveNet residual blocks, and HiFi-GAN’s generator and multi-period discriminator with specified modifications.
- Implementation details: The prior-encoder normalizing flow uses four volume-preserving affine coupling layers, each containing four WaveNet residual blocks.
- Implementation details: The posterior encoder uses 16 WaveNet residual blocks and produces 192-channel latent variables from linear-scale log-magnitude spectrograms.
B.2. Decoder and Discriminator
The decoder uses latent variables from prior or posterior encoders, while the discriminator is adapted from HiFi-GAN for efficient waveform adversarial training.
- The decoder receives latent variables generated by either the prior or posterior encoder, with an input channel size of 192.
- The final decoder convolution omits its bias parameter because the bias causes unstable gradient scales during mixed precision training.
- The discriminator retains HiFi-GAN’s five-period sub-discriminators and only the raw-waveform sub-discriminator from its multi-scale discriminator.
- Figure 5 depicts stochastic duration predictor training and inference, with a dilated and depth-wise separable convolutional residual block as its main building block.
B.3. Stochastic Duration Predictor
The stochastic duration predictor uses flow-based components and efficient DDSConv blocks to model duration distributions conditioned on text-related inputs.
- DDSConv residual blocks combine dilated and depth-wise separable convolutions with layer normalization and GELU activations.The design improves parameter efficiency while maintaining a large receptive field.
- The posterior encoder transforms Gaussian noise into ν and u for the approximate posterior qφ(u, ν|d, ctext).
- The normalizing flow transforms d −u and ν into Gaussian noise to express the log-likelihood of augmented, dequantized duration data.
- Condition encoders use two 1x1 convolutions and a DDSConv residual block, while the flow components contain four coupling layers of neural spline flows.
- Figure 6 presents the condition encoder and coupling-layer architectures used in the stochastic duration predictor.
C. Side-by-Side Evaluation
Side-by-side evaluations compare VITS with ground truth and examine speaker-independent voice conversion, including pitch behavior across target identities.
- -0.106 CMOS on LJ Speech and -0.270 CMOS on VCTK quantify VITS’s preference relative to ground truth in 500 ratings over 50 items.The evaluation used a 7-point Comparative Mean Opinion Score.
- VITS outperforms Glow-TTS with HiFi-GAN while remaining slightly less preferred than ground truth.
- Voice Conversion: In the multi-speaker setting, excluding speaker identities from the text encoder encourages speaker-independent latent representations for voice conversion.
- Voice Conversion: Voice conversion synthesizes a target-speaker voice by applying the inverse flow transformation to the speaker-independent representation and decoding it.
- Voice Conversion: The voice-conversion results provide raw waveforms and show similar pitch-track trends at different pitch levels across speaker identities.