Source-linked AI summary
FastDiff: A Fast Conditional Diffusion Model for High-Quality Speech Synthesis
Rongjie Huang, Max W. Y. Lam, Jun Wang, Dan Su, Dong Yu, Yi Ren, Zhou Zhao
TL;DR
FastDiff addresses the difficulty of deploying diffusion-based speech synthesis, whose iterative sampling can require hundreds or thousands of iterations and whose accelerated sampling may degrade quality. It uses adaptive time-aware convolutions and predicted noise schedules, and achieves high-quality fast synthesis while FastDiff-TTS simplifies text-to-waveform generation without intermediate features.
Problem
Diffusion speech synthesis remains difficult to deploy because reducing reverse iterations can degrade quality, while existing systems may require hundreds or thousands of iterations.
Method
FastDiff combines time-aware location-variable convolutions with diverse receptive fields and a noise schedule predictor, while FastDiff-TTS generates waveforms from phonemes without intermediate features.
Results
FastDiff outperformed the best publicly available models in synthesis quality, while FastDiff-TTS outperformed competing architectures; FastDiff achieved 58x faster than real-time sampling on a V100 GPU.
Takeaways & Limitations
FastDiff supports fast, high-fidelity diffusion-based speech synthesis, and FastDiff-TTS simplifies the end-to-end text-to-speech generation pipeline.
Abstract
from arXiv · showhide
Denoising diffusion probabilistic models (DDPMs) have recently achieved leading performances in many generative tasks. However, the inherited iterative sampling process costs hindered their applications to speech synthesis. This paper proposes FastDiff, a fast conditional diffusion model for high-quality speech synthesis. FastDiff employs a stack of time-aware location-variable convolutions of diverse receptive field patterns to efficiently model long-term time dependencies with adaptive conditions. A noise schedule predictor is also adopted to reduce the sampling steps without sacrificing the generation quality. Based on FastDiff, we design an end-to-end text-to-speech synthesizer, FastDiff-TTS, which generates high-fidelity speech waveforms without any intermediate feature (e.g., Mel-spectrogram). Our evaluation of FastDiff demonstrates the state-of-the-art results with higher-quality (MOS 4.28) speech samples. Also, FastDiff enables a sampling speed of 58x faster than real-time on a V100 GPU, making diffusion models practically applicable to speech synthesis deployment for the first time. We further show that FastDiff generalized well to the mel-spectrogram inversion of unseen speakers, and FastDiff-TTS outperformed other competing methods in end-to-end text-to-speech synthesis. Audio samples are available at \url{https://FastDiff.github.io/}.
1 Introduction
Speech synthesis requires both high fidelity and real-time speed, while diffusion models face costly iterative sampling. FastDiff addresses these challenges with adaptive convolutions, noise-schedule prediction, and an end-to-end waveform synthesizer.
- High-quality speech synthesis must reconstruct details across timescales, especially for high-sampling-rate waveforms.
- High generation speed is essential for real-time speech synthesis, challenging high-quality neural synthesizers.
- Hundreds to thousands of denoising steps can provide high sample quality, but reducing steps introduces perceivable background noise.
- FastDiff uses time-aware location-variable convolutions with diverse receptive fields to model long-term dependencies under adaptive conditions.
- FastDiff-TTS performs end-to-end phoneme-to-waveform synthesis without intermediate features or specialized loss functions.
- MOS 4.28 vs. 4.20 for WaveNet, while four iterations achieve 58x faster-than-real-time synthesis on a V100 GPU.
2 Background: Denoising Diffusion Probabilistic Models
DDPMs learn to reverse a fixed diffusion process from data to noise, enabling high-quality generative modeling. Their reverse sampling can require thousands of iterations, motivating more computationally efficient diffusion models.
- DDPMs are likelihood-based generative models that approximate a data distribution with a parameterized marginal distribution.
- The forward process gradually adds Gaussian noise through a fixed Markov chain, transforming data into whitened latent variables.
- The reverse process removes noise iteratively from Gaussian samples using a shared parameterized model to recover data samples.
- Reverse sampling may require up to thousands of iterative steps, motivating FastDiff's focus on reducing computational cost.
3 FastDiff
FastDiff combines time-aware location-variable convolutions with a learned noise schedule to improve conditional diffusion speech synthesis and accelerate sampling. FastDiff-TTS extends this approach to direct phoneme-to-waveform generation without intermediate acoustic features.
- 3 FastDiff: FastDiff uses time-aware location-variable convolutions with diverse receptive fields to model long-term dependencies in noisy, condition-dependent speech.The convolutions adapt to diffusion steps and acoustic features such as Mel-spectrograms.
- 3 FastDiff: The LVC module splits the input into segments and applies kernels generated by a predictor to associated sequence intervals.The kernel predictor produces variation-sensitive convolutional operations for each interval.
- 3 FastDiff: FastDiff’s noise schedule predictor derives a shorter sampling schedule than the training schedule, reducing reverse-diffusion iterations.Training uses T = 1000 discrete steps, while inference aligns a shorter schedule with N << T.
- 3.5 FastDiff-TTS: FastDiff-TTS directly produces waveforms from phoneme contexts without explicitly generating Mel-spectrograms or using specialized loss functions.Its architecture uses phoneme encoding, duration expansion, variance adaptation, waveform-segment training, and FastDiff decoding.
4 Related Works
Related speech-synthesis methods span autoregressive, flow-based, GAN-based, and direct text-to-waveform approaches. FastDiff-TTS differs by combining direct waveform generation with a fully differentiable, efficient diffusion architecture.
- 4 Related Works: WaveNet generates high-fidelity audio autoregressively but has prohibitively expensive computational costs.Its causal convolutions and large filters increase receptive fields while slowing inference.
- 4 Related Works: Flow-based and GAN-based models improve generation speed, but prior work reports limitations in sample quality or diversity.These approaches are presented as non-autoregressive alternatives to WaveNet.
- 4 Related Works: FastDiff-TTS directly generates waveforms without explicit intermediate spectrograms, specialized adversarial training, or the large footprint and slow inference associated with WaveGrad 2.The paper characterizes FastDiff-TTS as fully differentiable and efficient.
5 Experiments
Experiments evaluate FastDiff across quality, speed, diversity, unseen-speaker inversion, and end-to-end text-to-speech. FastDiff combines strong audio quality with four-step accelerated sampling, while FastDiff-TTS simplifies waveform synthesis without intermediate spectrograms.
- Experimental setup: The experiments use LJSpeech for reproducible comparison and VCTK to test generalization to unseen speakers.LJSpeech contains 13,100 clips from one female speaker; VCTK contains approximately 44,200 clips from 109 speakers.
- Experimental setup: FastDiff is compared with WaveNet, WaveGlow, HIFI-GAN V1, UnivNet, DiffWave, and WaveGrad using quality, speed, and diversity measures.Evaluation includes MOS, STOI, PESQ, real-time factor, NDB, and JSD.
- Comparison with other models: FastDiff achieved the highest MOS, matched WaveNet, outperformed non-autoregressive baselines, and showed a 0.24 gap from ground-truth audio.Objective evaluation also showed large improvements in PESQ and STOI.
- Comparison with other models: Four reverse steps produced high-fidelity speech and significantly reduced inference time compared with competing diffusion architectures.The noise predictor searches efficient schedules for accelerated sampling.
- Ablation study: The ablations show that time-aware location-variable convolutions improve speed and perceptual quality, while the noise predictor enables efficient sampling without sacrificing quality.Replacing the convolutions or using grid search degraded reported outcomes.
- Ablation study: Discrete diffusion-time conditioning synthesized higher-quality samples than continuous noise-level conditioning.The continuous variant does not require schedule alignment.
- Generalization to unseen speakers: FastDiff achieved state-of-the-art audio quality when inverting mel-spectrograms from five unseen VCTK speakers excluded from training.The evaluation used 50 randomly selected utterances.
- End-to-End Text-to-Speech: FastDiff-TTS surpassed competing end-to-end systems and matched state-of-the-art cascaded systems in voice quality.It simplifies text-to-speech by generating waveforms without the intermediate feature stage.
6 Conclusion
The conclusion presents FastDiff as a conditional diffusion model that combines adaptive long-term dependency modeling with predicted noise schedules for faster reverse sampling. Its FastDiff-TTS extension removes intermediate spectrograms, while experiments report high quality, few iterations, and unseen-speaker robustness.
- 6 Conclusion: FastDiff uses time-aware location-variable convolutions with diverse receptive fields to model long-term dependencies under adaptive conditions.The convolution design is intended to capture dynamic speech dependencies during diffusion.
- 6 Conclusion: A noise predictor derives tighter schedules that reduce reverse iterations without distinct quality degradation.The schedule is used to accelerate inference.
- 6 Conclusion: FastDiff-TTS discards intermediate features such as spectrograms and simplifies end-to-end text-to-waveform synthesis.The model extends FastDiff to fully end-to-end text-to-speech generation.
- 6 Conclusion: FastDiff generates high-quality samples in as few as 4 iterations and is reported as applicable to interactive, low-cost real-world speech generation.The conclusion describes synthesis quality as comparable to the human level.
- 6 Conclusion: FastDiff shows strong out-of-domain generalization and high-quality synthesis for unseen speakers.The conclusion summarizes robustness beyond the training speakers.
- Diffusion probabilistic models: The diffusion process gradually adds Gaussian noise according to a fixed schedule, converting data into whitened latents.The forward process is defined as a fixed Markov chain from x_0 to x_T.
- Diffusion probabilistic models: The reverse process recovers samples from Gaussian noise by iteratively eliminating the noise added during diffusion.It is parameterized by shared θ as a Markov chain from x_T to x_0.
- Acceleration: A learned scheduling network can replace the fixed schedule with an efficient N-step schedule for reverse sampling.The schedule predictor is trained using a KL-divergence objective between forward and reverse distributions.
B Model Architectures
FastDiff and FastDiff-TTS are documented through architecture hyperparameter tables, while Figure 3 visualizes the noise predictor and refinement-model DBlock. The FastDiff-TTS architecture follows a non-autoregressive text-to-speech backbone with duration and variance adaptation.
- FastDiff: Table 5 lists FastDiff’s architecture hyperparameters.
- FastDiff: Figure 3 visualizes the GALR-block-based noise predictor φ and the DBlock in FastDiff’s refinement model θ.
- FastDiff-TTS: Table 6 lists FastDiff-TTS’s architecture hyperparameters.
C Training, Noise scheduling and Inference details
FastDiff uses scheduled reverse diffusion with a noise-scheduling process and schedule alignment. The alignment interpolates between training noise constants to obtain floating-point diffusion steps for more efficient reverse sampling.
- Hyperparameters: Table 7 lists diffusion hyperparameters for FastDiff and FastDiff-TTS.
- Noise scheduling: FastDiff’s noise-scheduling algorithm follows bilateral denoising diffusion models.
- Noise scheduling: The scheduling process iterates backward, sampling x_t−1 from p_θ(x_t−1|x_t) and updating the schedule quantities.
- Schedule alignment: Schedule alignment searches and interpolates α_s between adjacent training noise constants l_t and l_t+1 to obtain the mapped diffusion step t_m.
- Schedule alignment: Floating-point t_m aligns the predefined diffusion schedule with the noise schedule for more efficient reverse sampling.
D Evaluation Matrix
The evaluation matrix uses speech-enhancement quality and intelligibility metrics alongside diversity measures. It also includes a crowd-sourced MOS evaluation whose testing interface is shown in Figure 4.
- Speech enhancement: PESQ and STOI assess denoising quality for speech enhancement.
- MOS evaluation: Figure 4 shows a screenshot of the MOS testing procedure.
- Diversity: NDB and JSD measure diversity by clustering training data and evaluating how generated samples fit those clusters.
D.3 Details in MOS Evaluation
MOS testing uses crowd-sourced ratings of synthetic-speech naturalness from native speakers. The section also describes continuous-noise-level conditioning and its separate training and sampling procedures.
- MOS evaluation: MOS tests are crowd-sourced, conducted by native speakers, and use a 1–5 Likert scale for sentence naturalness.Testers rate samples one at a time; participant compensation totaled about $750.
- MOS evaluation: Table 8 specifies the ratings used to evaluate synthetic and ground-truth speech naturalness.
- Continuous conditioning: The continuous-condition ablation removes the need for an additional schedule-alignment process.The passage describes schedule alignment as having separate training and sampling procedures.
- Continuous conditioning: Continuous-condition training samples t uniformly, interpolates α_s between adjacent schedule values, constructs x_s, and updates θ using noise-prediction loss.
F Sample Diversity
The section motivates evaluating speech diversity because diffusion models have shown greater sample diversity than GANs in image generation, while speech-domain comparisons remain overlooked. It examines whether diffusion models reduce mode collapse and generate diverse mel-spectrograms using NDB and JSD.
- Diffusion probabilistic models outperformed GANs in sample diversity for image generation, but speech-domain comparisons remain relatively overlooked.
- The study evaluates generated mel-spectrogram diversity with the NDB and JSD metrics.
- GAN-based models can suffer mode collapse, producing highly similar outputs from one or a few distribution modes in strongly conditional generation.
- Diffusion models are designed to reduce mode collapse through several conditional denoising diffusion steps rather than one-shot generation.