PaperlayerView exact revision on arXiv

Source-linked AI summary

Grad-TTS: A Diffusion Probabilistic Model for Text-to-Speech

Vadim Popov, Ivan Vovk, Vladimir Gogoryan, Tasnima Sadekova, Mikhail Kudinov

arXiv:2105.06337v2cs.LGcs.CLstat.ML

TL;DR

Diffusion-based acoustic feature generation lacks an established feature-generator approach despite successful diffusion vocoders. Grad-TTS uses a MAS-aligned, score-based decoder to transform encoder-parameterized Gaussian noise into mel-spectrograms with adjustable inference steps, and human evaluation finds it competitive with state-of-the-art TTS systems while the end-to-end variant remains below the reported quality.

  • Problem

    Diffusion probabilistic modelling had shown success for neural vocoding, but no diffusion-based acoustic feature generator was known.

  • Method

    Grad-TTS uses Monotonic Alignment Search and a score-based decoder to transform Gaussian noise parameterized by encoder outputs into mel-spectrograms, with inference speed controlled by decoder steps.

  • Results

    Grad-TTS-10 is competitive with all compared feature generators, including Tacotron2, while increasing reverse-diffusion iterations improves MOS with marginal gains beyond a certain point.

  • Takeaways & Limitations

    Grad-TTS can perform real-time synthesis and generate mel-spectrograms twice as fast as Tacotron2 while maintaining competitive synthesis quality.

  • Takeaways & Limitations

    The preliminary end-to-end TTS model produces fair-quality speech but cannot compete with the reported results.

Abstract

from arXiv · show

Recently, denoising diffusion probabilistic models and generative score matching have shown high potential in modelling complex data distributions while stochastic calculus has provided a unified point of view on these techniques allowing for flexible inference schemes. In this paper we introduce Grad-TTS, a novel text-to-speech model with score-based decoder producing mel-spectrograms by gradually transforming noise predicted by encoder and aligned with text input by means of Monotonic Alignment Search. The framework of stochastic differential equations helps us to generalize conventional diffusion probabilistic models to the case of reconstructing data from noise with different parameters and allows to make this reconstruction flexible by explicitly controlling trade-off between sound quality and inference speed. Subjective human evaluation shows that Grad-TTS is competitive with state-of-the-art text-to-speech approaches in terms of Mean Opinion Score. We will make the code publicly available shortly.

1. Introduction

Grad-TTS addresses the lack of diffusion-based acoustic feature generators by using a score-based decoder that transforms encoder-parameterized noise into mel-spectrograms. It provides controllable inference and achieves competitive quality with substantially faster generation in the reported setting.

  • Motivation: Parallel and autoregressive TTS systems improve naturalness, speed, or robustness, but attention failures and computational inefficiency remain issues for some feature generators.The related systems include Tacotron2, Transformer-TTS, FastSpeech, Parallel Tacotron, Glow-TTS, and several parallel vocoders.
  • Motivation: Diffusion vocoders such as WaveGrad and DiffWave match strong autoregressive baselines in speech quality, but no diffusion-based feature generator was known.The gap motivates applying diffusion probabilistic modelling to acoustic feature generation rather than only waveform vocoding.
  • Grad-TTS: Grad-TTS uses a score-based decoder to transform Gaussian noise parameterized by encoder outputs into mel-spectrograms.The encoder outputs are aligned with text using Monotonic Alignment Search.
  • Grad-TTS: The model explicitly controls the trade-off between mel-spectrogram quality and inference speed through its diffusion reconstruction procedure.This flexibility follows from the diffusion framework used for reconstructing data from Gaussian noise with varying parameters.
  • Results: Grad-TTS can generate high-quality mel-spectrograms with as few as ten reverse-diffusion iterations and outperform Tacotron2 in GPU speed.The paper also reports an end-to-end variant that combines feature generation and vocoding by producing raw waveforms.

2. Diffusion probabilistic modelling

Diffusion probabilistic modelling constructs a forward process that converts data into Gaussian noise and a learned reverse process that reconstructs the data. Grad-TTS uses an SDE formulation generalized to terminal Gaussian distributions with specified mean and covariance, with score estimation enabling reverse sampling.

  • Diffusion framework: A diffusion-type process is defined by a stochastic differential equation with drift and diffusion coefficients driven by Brownian motion.The coefficients must satisfy measurability conditions for the process formulation used in the paper.
  • Forward diffusion: Forward diffusion transforms any initial data distribution into a simple terminal distribution, typically standard normal, as time approaches infinity.The reverse-diffusion goal is to follow the forward trajectories in reverse time.
  • Generalized diffusion: The paper generalizes the framework so forward diffusion converges to N(µ, Σ), with specified mean µ and diagonal covariance matrix Σ, rather than only N(0, I).This generalization is designed for reconstructing data from noise with varying parameters.
  • Forward diffusion: The proposed forward SDE produces a conditional Gaussian distribution, and its state converges in distribution to N(µ, Σ) independently of the initial data.This establishes the Gaussian terminal condition required by the generalized diffusion framework.
  • Reverse diffusion: Reverse diffusion is solved backward from the terminal Gaussian distribution using dynamics derived from the forward process and the noisy-data score.The score network estimates the gradient of the log-density, allowing backward numerical sampling through either the reverse SDE or an equivalent ODE.
  • Score estimation: Score matching trains a neural network to estimate noisy-data log-density gradients using an L2 loss.Because the conditional noisy-data distribution is Gaussian, the training formulation can sample noisy states directly without simulating intermediate states.

3. Grad-TTS

Grad-TTS combines text alignment, duration prediction, and a diffusion-based decoder to generate mel-spectrograms from text-conditioned Gaussian noise. Its reverse-diffusion solver and noise parameterization support adjustable inference speed and quality.

  • Architecture and alignment: The encoder and duration predictor produce a hard monotonic alignment that expands text features into frame-wise conditioning features.Monotonicity and surjectiveness preserve pronunciation order and ensure every text element is represented.
  • Architecture and alignment: The decoder transforms terminal Gaussian noise conditioned on aligned features into mel-spectrograms using a score-based diffusion model.The terminal condition is sampled as N(µ, I), where µ is the aligned encoder output.
  • Inference: The Euler step size h is selected after training to control the trade-off between mel-spectrogram quality and inference speed.The reverse-time ODE is solved with a first-order Euler scheme.
  • Inference: Larger step sizes produced better sounding results with dynamics (9) than with dynamics (8), while both performed equally for small step sizes.This comparison motivated the reverse-diffusion dynamics used in Grad-TTS.
  • Inference: The model additionally uses the conditioning mean µ as an input to the score network and a temperature τ to preserve quality at larger step sizes.Sampling from N(µ, τ^-1I) allows temperature tuning to compensate for larger Euler steps.
  • Training: The encoder loss is a negative log-likelihood equivalent to mean squared error, and removing it caused Grad-TTS to fail to learn alignment in practice.The diffusion loss instead trains the score network to estimate noisy-data log-density gradients across times t ∈[0, T].
  • Training: Training alternates between Monotonic Alignment Search and parameter optimization for the encoder, duration predictor, and diffusion objectives.The encoder loss is optimized jointly with alignment through iterative alignment search followed by stochastic gradient updates.
  • Architecture: The decoder uses a reduced U-Net architecture operating on 80-dimensional mel-spectrograms, while the encoder and duration predictor follow Glow-TTS designs.The decoder uses three feature-map resolutions and fewer channels to reduce model size.

4. Experiments

Experiments evaluate Grad-TTS on LJSpeech through subjective quality, generalized-diffusion ablation, error analysis, likelihood, efficiency, and preliminary end-to-end synthesis. The results show competitive quality, flexible speed-quality control, favorable likelihood and parameter efficiency, while the end-to-end variant remains below the main model.

  • Experimental setup: Grad-TTS was trained on approximately 24 hours of LJSpeech English female speech using phonemized text and 80-dimensional mel-spectrograms.The test set contained around 500 short recordings, each under 10 seconds.
  • Subjective evaluation: Varying decoder steps N enables explicit control of the quality-speed trade-off, with models evaluated at N=4, 10, 100, and 1000.All four models used τ = 1.5 at synthesis and were compared with Glow-TTS, FastSpeech, and Tacotron2 using HiFi-GAN.
  • Subjective evaluation: Grad-TTS-10 is competitive with all compared feature generators, while Grad-TTS-1000 achieves nearly natural synthesis with MOS only 0.1 below ground-truth recordings.Increasing reverse-diffusion iterations improves MOS, but gains become marginal after a certain number; Grad-TTS-1000 and Grad-TTS-10 show almost no MOS difference.
  • Ablation study: The generalized framework outperformed reconstruction from N(0, I): Grad-TTS-10 was significantly better even against models using 10, 20, or 50 reverse-diffusion iterations.The reported preference test found p < 0.005 in a sign test; the authors attribute the difference to reconstructing from N(µ, I) rather than pure noise.
  • Objective evaluation: Grad-TTS achieved better average log-likelihood than Glow-TTS despite Glow-TTS having three times larger decoder capacity and being trained for exact data likelihood.The comparison used 50 randomly selected test sentences, with Grad-TTS likelihood reported using a 95% confidence interval enabled by Hutchinson’s trace estimator.
  • Efficiency estimation: On GPU, Grad-TTS reaches RTF < 0.37 with fewer than 100 decoder steps and is approximately twice faster than Tacotron2 at 10 steps, while using around 15m parameters.The authors note that Grad-TTS remains slower than Glow-TTS and FastSpeech.
  • End-to-end TTS: Preliminary end-to-end DPM training produced fair-quality speech but did not match the main Grad-TTS results and was excluded from the listening test.The end-to-end decoder was adapted from WaveGrad and conditioned on aligned encoder output rather than ground-truth mel-spectrograms.

5. Future work

The paper identifies open questions about end-to-end diffusion-based TTS and broader practical choices in diffusion probabilistic models. Future work includes improving end-to-end synthesis and investigating noise schedules and loss weighting.

  • Future directions: End-to-end diffusion-based speech synthesis is a promising direction for TTS applications but still has substantial room for improvement.The preliminary end-to-end model produced fair-quality speech but did not match the reported results of the main model.
  • Diffusion-model theory: The analysis assumes that the forward and reverse diffusion SDEs have strong solutions.This assumption imposes Lipschitz-type constraints on the noise schedule β_t and the neural network s_θ.
  • Practical diffusion design: The choice of noise schedule β_t and weights for time-dependent losses remains insufficiently understood.The paper notes that most researchers use a simple linear noise schedule and that optimal loss weighting is unclear.

6. Conclusion

Grad-TTS is introduced as a diffusion-based acoustic feature generator that aligns text with mel-spectrograms and transforms parameterized Gaussian noise into speech features. It supports adjustable inference steps, real-time synthesis, and competitive quality while generating mel-spectrograms twice as fast as Tacotron2.

  • Conclusion: Grad-TTS is the first acoustic feature generator presented in the paper to use diffusion probabilistic modelling.Its diffusion-based decoder transforms Gaussian noise parameterized by encoder output into mel-spectrograms, with Monotonic Alignment Search for alignment.
  • Conclusion: Varying the number of decoder steps at inference controls the trade-off between synthesis quality and inference speed.The iterative decoder remains capable of real-time synthesis.
  • Conclusion: Grad-TTS generates mel-spectrograms twice faster than Tacotron2 while maintaining synthesis quality competitive with common TTS baselines.This conclusion summarizes the paper’s reported speed and quality comparison.

Appendix

The appendix develops the generalized diffusion framework used by Grad-TTS. It replaces the conventional standard-normal terminal distribution with a Gaussian distribution parameterized by encoder-derived mean and covariance.

  • Appendix: Grad-TTS uses a generalized terminal distribution N(µ, Σ) instead of the conventional N(0, I).The appendix provides derivations for the forward SDE solution and the distribution of the diffused process X_t.

A. Solving forward diffusion SDE

The forward diffusion SDE is formulated for an n-dimensional stochastic process with Gaussian parameters and a non-negative noise schedule. A change of variables centers the process by µ and enables an equivalent transformed formulation.

  • A. Solving forward diffusion SDE: The forward diffusion SDE uses X_t, standard Brownian motion W_t, an n-dimensional mean vector µ, and a diagonal covariance matrix Σ.The covariance matrix has positive diagonal elements, while the noise schedule β_t is non-negative.
  • A. Solving forward diffusion SDE: Centering the process with Y_t = X_t − µ rewrites the forward diffusion SDE in terms of the shifted variable Y_t.The transformed formulation separates the process from its Gaussian mean parameter.
  • A. Solving forward diffusion SDE: Because Σ is diagonal, its matrix exponential can be expressed as element-wise exponentials in the multidimensional formulation.The appendix then rewrites the transformed solution and returns to the original variable X_t.
  • A. Solving forward diffusion SDE: The identity matrix I specifies the isotropic component of the multidimensional formulation.The appendix identifies I as an n × n identity matrix when expressing the solution in terms of X_t.

B. Derivation of conditional distribution of Xt

The derivation establishes the conditional distribution of the diffusion process as Gaussian, with parameters determined by the initial state and diffusion coefficients. It extends the result to the multidimensional case using properties of Brownian motion and subsequent integration.

  • The diffusion matrix is diagonal, with diagonal element aii(s) equal to √βs.
  • Each stochastic integral is defined as the limit of integral sums as the partition mesh tends to zero.
  • The integral-sum increments are independent normal variables because the coefficients are deterministic and the Brownian increments are independent.
  • Lévy’s continuity theorem supports the distributional limit, after which direct integration yields the conditional process distribution.
  • In multiple dimensions, the conditional law is Gaussian with mean ρ(X0, Σ, µ, t) and covariance λ(Σ, t).

C. Reverse dynamics

The paper derives reverse-time diffusion dynamics involving the score ∇log pt(Xt), and relates the reverse SDE to an equivalent ODE under shared forward Kolmogorov dynamics. These equations are solved backward in time.

  • Anderson’s result gives reverse-time dynamics for an n-dimensional diffusion process satisfying the stated forward SDE.
  • The reverse SDE has drift f(Xt, t) − g^2(t)∇log pt(Xt) and reverse-time Brownian noise g(t)dẆt.
  • Reverse-time differentials use t as the lower integration limit, with the reverse Brownian motion independent of its past increments.
  • The SDE and a corresponding ODE share the same forward Kolmogorov equation and therefore have equal distributions when initialized identically.
  • For the considered process, substituting f(Xt, t) = 1/2Σ−1(Xt − µ)βt and g(t) = √βt yields two equivalent reverse diffusion dynamics.
  • Both reverse differential equations are solved backward in time.

D. Score estimation

The score-estimation derivation conditions on a known initial state and expresses the noisy sample through its conditional Gaussian distribution. In the isotropic case, the conditional score becomes a scaled negative noise term.

  • When X0 is known, the conditional density p0t(·|X0) is the Gaussian density associated with Law(Xt|X0).
  • Sampling Xt as ρ(X0, Σ, µ, t) + ϵt with ϵt ∼ N(0, λ(Σ, t)) gives score ∇log p0t(Xt|X0) = −λ(Σ, t)−1ϵt.
  • With Σ = I, the covariance simplifies to λ(I, t) = λtI, where λt = 1 − e−∫0^t βsds.
  • Under Σ = I, the noisy-data log-density gradient reduces to ∇log p0t(Xt|X0) = −ϵt/λt.
Loading 2105.06337v2…