PaperlayerView exact revision on arXiv

Source-linked AI summary

ProDiff: Progressive Fast Diffusion Model For High-Quality Text-to-Speech

Rongjie Huang, Zhou Zhao, Huadai Liu, Jinglin Liu, Chenye Cui, Yi Ren

arXiv:2207.06389v1eess.AScs.LGcs.SD

TL;DR

Diffusion TTS systems require hundreds or thousands of denoising iterations for high-quality synthesis, limiting deployment. ProDiff predicts clean data directly and uses knowledge distillation to halve the teacher’s steps. It synthesizes high-fidelity mel-spectrograms in two iterations while retaining competitive quality and diversity and reaching 24x faster-than-real-time sampling.

  • Problem

    Diffusion TTS models require hundreds or thousands of iterations for high sample quality, hindering practical deployment.

  • Method

    ProDiff directly predicts clean data and distills an N-step DDIM teacher into a new model with N/2 steps to reduce target variance.

  • Results

    ProDiff needs only 2 iterations for high-fidelity mel-spectrograms while maintaining sample quality and diversity competitive with state-of-the-art models using hundreds of steps.

  • Takeaways & Limitations

    ProDiff makes diffusion models applicable to interactive, real-world text-to-speech at low computational cost and extends readily to multi-speaker synthesis.

  • Takeaways & Limitations

    The model is described as dividing the diffusion process into conditional steps expected to improve training stability and mode coverage.

Abstract

from arXiv · show

Denoising diffusion probabilistic models (DDPMs) have recently achieved leading performances in many generative tasks. However, the inherited iterative sampling process costs hinder their applications to text-to-speech deployment. Through the preliminary study on diffusion model parameterization, we find that previous gradient-based TTS models require hundreds or thousands of iterations to guarantee high sample quality, which poses a challenge for accelerating sampling. In this work, we propose ProDiff, on progressive fast diffusion model for high-quality text-to-speech. Unlike previous work estimating the gradient for data density, ProDiff parameterizes the denoising model by directly predicting clean data to avoid distinct quality degradation in accelerating sampling. To tackle the model convergence challenge with decreased diffusion iterations, ProDiff reduces the data variance in the target site via knowledge distillation. Specifically, the denoising model uses the generated mel-spectrogram from an N-step DDIM teacher as the training target and distills the behavior into a new model with N/2 steps. As such, it allows the TTS model to make sharp predictions and further reduces the sampling time by orders of magnitude. Our evaluation demonstrates that ProDiff needs only 2 iterations to synthesize high-fidelity mel-spectrograms, while it maintains sample quality and diversity competitive with state-of-the-art models using hundreds of steps. ProDiff enables a sampling speed of 24x faster than real-time on a single NVIDIA 2080Ti GPU, making diffusion models practically applicable to text-to-speech synthesis deployment for the first time. Our extensive ablation studies demonstrate that each design in ProDiff is effective, and we further show that ProDiff can be easily extended to the multi-speaker setting. Audio samples are available at \url{https://ProDiff.github.io/.}

1 INTRODUCTION

Diffusion TTS models offer high-quality and diverse synthesis but face a deployment trade-off: many denoising steps are needed for quality, while fewer steps can blur predictions. ProDiff addresses this with clean-data prediction and teacher distillation, achieving high-fidelity synthesis in two iterations.

  • Motivation: High-quality TTS must capture fine speech details, operate quickly, and preserve diversity to avoid dull or unimodal long-form speech.The stated goals include modeling harmonic gaps, unvoiced frames, and high-frequency components while supporting real-time generation.
  • Challenges: Hundreds to thousands of denoising steps are typically required for DDPMs to guarantee high sample quality, hindering real-world deployment.This iterative cost is a central challenge for diffusion-based speech synthesis.
  • Challenges: Reducing diffusion iterations can degrade convergence and produce blurry, over-smooth mel-spectrogram predictions.The degradation is attributed to the complex data distribution.
  • ProDiff: ProDiff directly predicts clean data and distills an N-step DDIM teacher into a student using N/2 steps.Knowledge distillation reduces target variance, enabling sharper predictions while accelerating sampling by orders of magnitude.
  • Results: 2 iterations synthesize high-fidelity mel-spectrograms while maintaining sample quality and diversity competitive with state-of-the-art models using hundreds of steps.The reported result is accompanied by a claimed practical deployment benefit for diffusion TTS.

2 BACKGROUND ON DIFFUSION MODELS

Diffusion probabilistic models transform data into whitened latent variables through a fixed noisy forward process, then recover samples through a learned reverse Markov chain. Their reverse sampling can require thousands of iterations, motivating faster conditional diffusion models.

  • Diffusion process: Diffusion models define data as q(x_0) and use a fixed Markov chain to transform it into a latent variable x_T.The forward process progressively adds Gaussian noise according to a fixed schedule.
  • Diffusion process: A small Gaussian noise is added at each forward step under q(x_t|x_{t-1}).The noise schedule gradually converts x_0 into whitened latents x_T.
  • Reverse process: The reverse process recovers samples from Gaussian noise through a θ-parameterized Markov chain from x_T to x_0.Each iteration removes Gaussian noise introduced during the diffusion process.
  • Motivation: Thousands of reverse-sampling steps can be required to reconstruct the target distribution, creating a computational-efficiency problem.The paper presents a progressive fast conditional diffusion model to reduce reverse iterations.

3 DIFFUSION MODEL PARAMETERIZATION

Diffusion TTS models can parameterize denoising either through gradient estimation or direct clean-data prediction. The generator-based formulation avoids gradient estimation and is better suited to accelerating sampling from complex distributions.

  • Gradient-based method: Gradient-based diffusion models learn data-density gradients or noise-related representations and typically optimize mean squared error in 𝝐-space.The denoising model estimates the Stein score or directly predicts 𝝐, with training performed over randomly selected diffusion times.
  • Gradient-based method: Hundreds to thousands of denoising steps are typically required for high sample quality, hindering real-world text-to-speech deployment.
  • Generator-based method: Generator-based diffusion models directly predict clean data 𝒙0 rather than estimating the gradient of data density.
  • Generator-based method: The generator-based model predicts unperturbed 𝒙0 and samples 𝒙t−1 using q(𝒙t−1|𝒙t, 𝒙0), avoiding direct prediction of 𝒙t−1 across perturbation levels.
  • Generator-based method: Generator-based methods use mean squared error in data 𝒙-space and show advantages in accelerating sampling from complex distributions.

4 PRODIFF

ProDiff combines generator-based clean-data prediction with knowledge distillation to reduce diffusion iterations while preserving high-quality speech synthesis. Its architecture builds on FastSpeech 2 and integrates phoneme, variance, and spectrogram-denoising components.

  • 4.1 Motivation: ProDiff addresses diffusion TTS deployment challenges by combining generator-based parameterization with knowledge distillation.The model directly predicts clean data and trains a reduced-variance target from an N-step DDIM teacher for an N/2-step student.
  • 4.2 Select a teacher: A 4-step generator-based diffusion model is selected as the teacher because it balances perceptual quality, speed, and diversity.
  • 4.3 Knowledge distillation: The student is initialized from the teacher, uses two teacher DDIM steps as its target, and matches each student DDIM step to two teacher steps.
  • 4.4 Model architecture: ProDiff comprises a phoneme encoder, variance adaptor, and spectrogram denoiser built upon FastSpeech 2.The variance adaptor predicts duration, pitch, and energy before denoising the spectrogram.
  • 4.5 Training Loss: Training combines sample reconstruction, SSIM, and variance reconstruction losses for mel-spectrogram, duration, pitch, and energy targets.Variance losses use target duration, energy, and pitch values with corresponding predictions.
  • 4.6 Inference: During inference, the denoiser predicts 𝒙0, posterior sampling produces 𝒙t−1, and a pretrained vocoder converts the final spectrogram into waveforms.

5 RELATED WORKS

Prior TTS work progressed from autoregressive synthesis to parallel mel-spectrogram generation, while diffusion TTS retained high quality at the cost of iterative sampling. ProDiff differs by targeting efficient conditional diffusion without unstable adversarial training.

  • 5.1 Text-to-Speech: Autoregressive TTS models generate samples sequentially and therefore have slow inference, whereas non-autoregressive models generate mel-spectrogram frames in parallel.
  • 5.1 Text-to-Speech: Diff-TTS, Grad-TTS, and DiffSpeech use gradient-based score-matching objectives, but iterative sampling limits their text-to-speech applications.
  • 5.2 Diffusion Probabilistic Models: Diffusion models produce high-quality samples across image, audio, and 3D point-cloud domains, but their many reverse-sampling steps reduce efficiency.
  • 5.2 Diffusion Probabilistic Models: Prior acceleration methods focus mainly on image diffusion, while audio synthesis involves long-term dependencies and strong conditioning.
  • 5.3 Knowledge Distillation: Knowledge distillation has been used to simplify data distributions and address one-to-many mappings in non-autoregressive translation and text-to-speech.
  • 5.3 Knowledge Distillation: ProDiff uses discrete-schedule distillation to reduce data variance and improve convergence while avoiding unstable adversarial learning.

6 EXPERIMENTS

Experiments evaluate ProDiff against diffusion and non-autoregressive TTS systems on quality, speed, diversity, robustness, and component effectiveness. ProDiff preserves high-quality and diverse synthesis while reducing diffusion sampling to two iterations.

  • Experimental Setup: ProDiff was evaluated on LJSpeech using quality, latency, and diversity comparisons against autoregressive, non-autoregressive, flow-based, and diffusion TTS systems.The evaluation used 13,100 clips from a female speaker, totaling about 24 hours, with HiFi-GAN as vocoder and human MOS evaluation.
  • Preliminary Analyses: Gradient-based models degrade distinctly at T≤16 steps, whereas generator-based models maintain better quality when iterative sampling is reduced.With large noise schedules, both parameterizations can synthesize high-fidelity speech with similar results.
  • Performances: ProDiff achieves high perceptual quality, with a 0.17 gap compared to ground-truth audio, and outperforms baseline models on MCD, PESQ, and STOI.It matches state-of-the-art DDPMs using hundreds of steps and outperforms other non-autoregressive baselines.
  • Performances: 2 iterations and 24x faster than real-time enable ProDiff to synthesize high-fidelity spectrograms on a single NVIDIA 2080Ti GPU.Its latency remains nearly constant at 20ms as phoneme-sequence length increases, unlike systems whose latency grows linearly.
  • Sample Diversity: ProDiff achieves higher NDB and JS diversity scores than several one-shot methods, while GAN-based synthesis shows minimal diversity.Visualizations associate diffusion models with richer frequency details and more natural, expressive sounds than non-probabilistic and GAN-based systems.
  • Robustness and Multi-Speaker Extension: ProDiff obtains comparable robustness to state-of-the-art non-autoregressive TTS models and extends to multi-speaker evaluation with similar conclusions.The robustness evaluation covers both single-speaker and multi-speaker datasets.
  • Ablation Studies: Ablations show that generator parameterization prevents quality degradation at few steps, while knowledge distillation reduces variance and promotes convergence.Removing distillation produces blurry, over-smooth predictions; distilling from a 4-step teacher offers the reported balance between cost and quality.

7 CONCLUSION

ProDiff addresses the slow sampling and convergence challenges of diffusion TTS by predicting clean data and distilling an N-step teacher into an N/2-step student. Experiments show two-step synthesis with competitive quality and diversity.

  • 7 CONCLUSION: ProDiff targets the hundreds of iterations required by gradient-based diffusion TTS to preserve high sample quality.The approach is motivated by the challenge of accelerating reverse diffusion without significant perceptual degradation.
  • 7 CONCLUSION: ProDiff directly predicts clean data and uses teacher-generated mel-spectrograms as targets to reduce target variance and sharpen predictions.Knowledge distillation transfers an N-step teacher's behavior to a model using N/2 steps.
  • 7 CONCLUSION: 2 iterations synthesize high-fidelity mel-spectrograms while maintaining sample quality and diversity competitive with state-of-the-art models using hundreds of steps.The conclusion also reports effective ablations and extension to a multi-speaker setting.

C ARCHITECTURE

The architecture section lists ProDiff's model hyperparameters and training algorithms for gradient-based and generator-based diffusion models. The supplied passages identify the relevant algorithm components but do not provide their full specifications.

  • C ARCHITECTURE: Table 4 lists the hyperparameters of the ProDiff models.No individual hyperparameter values are provided in the supplied passage.
  • C ARCHITECTURE: Algorithm 3 is the training procedure for the gradient-based diffusion model.The supplied algorithm fragment samples clean data, Gaussian noise, and a diffusion timestep.
  • C ARCHITECTURE: The listed training objectives include LSSIM, Lp, Le, and Ldur terms alongside the diffusion-model objective.The supplied passages do not define the individual loss terms.
  • C ARCHITECTURE: Algorithm 4 is the training procedure for the generator-based diffusion model.Its supplied fragment likewise requires a diffusion model and samples data, noise, and a timestep.

E RESULTS ON MULTI-SPEAKER DATASET

ProDiff is evaluated on a multi-speaker LibriTTS setting and achieves strong subjective and objective audio quality in this more complicated scenario.

  • The multi-speaker evaluation uses the train-clean-100 LibriTTS subset, containing 247 speakers and about 54 hours of recordings.
  • ProDiff achieves improved quality in both subjective and objective evaluation on the multi-speaker dataset.
  • Table 5 reports audio performance comparisons for multi-speaker scenarios.

F ROBUSTNESS EVALUATION

Robustness is tested on difficult sentences from both single-speaker and multi-speaker datasets, where ProDiff performs comparably to state-of-the-art non-autoregressive TTS models.

  • ProDiff achieves comparable robustness performance with state-of-the-art non-autoregressive TTS models.
  • The robustness evaluation covers both the LJSpeech and LibriTTS datasets.
  • The evaluation selects 50 sentences that are particularly hard for TTS systems.
  • Tables 6 and 7 report robustness results for LJSpeech and LibriTTS, respectively.

G.1 Subjective Evaluation

Subjective evaluation uses crowdsourced native-speaker MOS testing, while the broader evaluation includes spectral, speech-enhancement, diversity, and spectrogram-visualization analyses.

  • MOS tests are crowdsourced and conducted by native speakers, with samples presented and rated one at a time.
  • Table 8 provides the ratings used to evaluate the naturalness of synthetic and ground-truth speech samples.
  • MCD measures spectral distance between synthesized and reference mel-spectrum features.
  • PESQ and STOI assess denoising quality for speech enhancement.
  • NDB and JSD measure generated-sample diversity by comparing cluster assignments with the training data.
  • Figure 4 visualizes ground-truth and generated mel-spectrograms from TTS models using different diffusion-step counts.
Loading 2207.06389v1…