Diff-TTS: A Denoising Diffusion Model for Text-to-Speech

Myeonghun Jeong; Hyeongju Kim; Sung Jun Cheon; Byoung Jin Choi; Nam Soo Kim

Source-linked AI summary

Diff-TTS: A Denoising Diffusion Model for Text-to-Speech

Myeonghun Jeong, Hyeongju Kim, Sung Jun Cheon, Byoung Jin Choi, Nam Soo Kim

arXiv:2104.01409v1Apr 3, 2021eess.AScs.AIcs.SD

TL;DR

Neural TTS still faces challenges in naturalness, inference efficiency, and architectural limitations. Diff-TTS addresses these issues with a non-autoregressive denoising diffusion model, likelihood-based training, and accelerated sampling. It achieves high-fidelity synthesis with fewer parameters and inference up to 28 times faster than real time.

Problem
Neural TTS models still require improvements in naturalness and architectural efficiency, while autoregressive systems have inference-time and robustness limitations and flow-based systems can be parameter-inefficient.
Source: §1 · p. 1
Method
Diff-TTS transforms Gaussian noise into text-conditioned mel-spectrograms through denoising diffusion, trains with a log-likelihood objective, and uses accelerated sampling for faster synthesis.
Source: §1 · p. 1, §2.1 · p. 2, §2.2 · p. 2
Results
28 times faster than real time, Diff-TTS achieves faster-than-real-time inference while maintaining higher MOS than Tacotron2 at γ = 57.
Source: §3.2 · p. 4
Takeaways & Limitations
Diff-TTS provides controllable, high-fidelity non-AR speech synthesis with fewer parameters and a user-adjustable quality–speed trade-off.
Source: §1 · p. 1, §4 · p. 4
Flow-based TTS models are described as parameter-inefficient because of architectural constraints imposed by normalizing flows.
Source: §1 · p. 1

Abstract

from arXiv · show

Although neural text-to-speech (TTS) models have attracted a lot of attention and succeeded in generating human-like speech, there is still room for improvements to its naturalness and architectural efficiency. In this work, we propose a novel non-autoregressive TTS model, namely Diff-TTS, which achieves highly natural and efficient speech synthesis. Given the text, Diff-TTS exploits a denoising diffusion framework to transform the noise signal into a mel-spectrogram via diffusion time steps. In order to learn the mel-spectrogram distribution conditioned on the text, we present a likelihood-based optimization method for TTS. Furthermore, to boost up the inference speed, we leverage the accelerated sampling method that allows Diff-TTS to generate raw waveforms much faster without significantly degrading perceptual quality. Through experiments, we verified that Diff-TTS generates 28 times faster than the real-time with a single NVIDIA 2080Ti GPU.

1. Introduction

Diff-TTS is a non-autoregressive TTS model that applies denoising diffusion to improve synthesis quality and efficiency while avoiding architectural constraints of flow-based models. It combines likelihood-based training, accelerated sampling, and prosody control.

Scope

The model targets acoustic modeling by converting text into mel-spectrograms before waveform generation by a vocoder.The paper focuses on the acoustic-modeling component of a standard neural TTS pipeline.Source: §1 · p. 1

Contributions

Diff-TTS uses log-likelihood-based optimization to train without auxiliary loss functions or model-architecture constraints.The denoising diffusion framework supports stable optimization while preserving freedom in architecture choices.Source: §1 · p. 1
Diff-TTS applies a denoising diffusion probabilistic model to non-AR mel-spectrogram generation.The model is presented as the first DDPM applied to non-AR TTS.Source: §1 · p. 1
Diff-TTS achieves high-fidelity synthesis with half the parameters of Tacotron2 and Glow-TTS, measured by Mean Opinion Score.This addresses efficiency alongside synthesis quality.Source: §1 · p. 1
Accelerated sampling lets users trade sample quality against inference speed according to available computational resources.The method reduces reverse transitions without significantly degrading sample quality and achieves faster-than-real-time inference.Source: §1 · p. 1
Diff-TTS controls pitch variability through latent-space and additive-noise variances scaled by a temperature term.The paper analyzes this mechanism as a form of speech-prosody control.Source: §1 · p. 1

2. Diff-TTS

Diff-TTS uses a denoising diffusion process to generate text-conditioned mel-spectrograms, with reverse transitions recovering speech features from Gaussian noise. Accelerated sampling reduces reverse transitions while preserving high-fidelity mel-spectrogram generation, supported by a text-conditioned architecture.

Denoising diffusion model for TTS

Diff-TTS transforms mel-spectrograms into Gaussian noise through Markov diffusion transitions and reverses this process to generate text-conditioned mel-spectrograms.The forward process uses a predefined variance schedule, while the reverse process restores mel-spectrograms from noise.Source: §2.1 · p. 2
The reverse process maximizes the conditional mel-spectrogram log-likelihood, using an L1 loss between predicted and sampled Gaussian noise without auxiliary losses.During inference, the model predicts the noise added at each forward transition and removes the corrupted component iteratively.Source: §2.1 · p. 2

Accelerated Sampling

Accelerated sampling selects a shorter reverse-time subsequence using decimation factor γ, reducing reverse transitions without model retraining or significant sample-quality deterioration.The new reverse path periodically selects transitions from the original trajectory.Source: §2.2 · p. 2
Accelerated sampling produces high-fidelity mel-spectrograms over the subsequence τ and increases inference speed for mel-spectrogram generation.This method is intended to improve synthesis speed while preserving sample quality.Source: §2.2 · p. 2

Model Architecture

The Diff-TTS architecture combines text encoding, diffusion-step encoding, duration prediction, length regulation, and decoding to produce Gaussian noise conditioned on phoneme sequences and diffusion time-steps.The encoder provides contextual phoneme information, while the length regulator expands phoneme embeddings to match mel-spectrogram sequence length.Source: §2.3 · p. 3

3. Experiments

Experiments evaluate Diff-TTS for audio quality, model size, inference speed, and prosody controllability. The results show high-quality synthesis with fewer parameters, substantially faster-than-real-time generation, and temperature-controlled prosodic diversity.

Audio quality and model size

Diff-TTS achieved high-fidelity audio while requiring about 53% fewer parameters than Tacotron2 and Glow-TTS.The model required the fewest parameters among the compared systems.Source: §3.1 · p. 3
Accelerated sampling preserved speech quality comparably to Tacotron2 and Glow-TTS, with degradation at γ = 57 reported as not significant.The best MOS occurred with 400 diffusion time steps, while accelerated settings remained practical for synthesis.Source: §3.1 · p. 3

Inference speed

Accelerated sampling gave Diff-TTS an RTF of 0.035 at γ = 57 while maintaining higher MOS than Tacotron2.RTF measures the time required to synthesize one second of waveform; evaluations used 500 test sentences on one NVIDIA 2080Ti GPU.Source: §3.1 · p. 3, §3.2 · p. 4, Table 3 · p. 4
Diff-TTS ran 28 times faster than real-time, although Glow-TTS remained faster on the same comparison.The accelerated sampler enables a quality–speed trade-off based on available computational resources.Source: §3.2 · p. 4

Variability and controllability

Increasing the temperature term η from 0.2 to 0.6 produced more diverse speech while maintaining speech quality.Temperature scaling modifies latent-space and additive-noise variances, allowing prosody variability to be controlled without retraining.Source: §3.3 · p. 4, Figure 4 · p. 4

4. Conclusions

Diff-TTS is presented as a controllable, high-fidelity non-autoregressive diffusion model for mel-spectrogram generation, with improved quality, efficiency, speed control, and prosodic variability.

Diff-TTS outperforms Tacotron2 and Glow-TTS in synthesis quality with fewer parameters.The model achieves high-fidelity audio according to Mean Opinion Score while using half the parameters of both systems.Source: §4 · p. 4
Accelerated sampling lets users trade sample quality for inference speed, reaching up to 28 times faster than real-time without significantly degrading audio quality.Source: §4 · p. 4
Multiplying a temperature term gives Diff-TTS control over speech prosodic variability.Source: §4 · p. 4
Diff-TTS is identified as the first diffusion generative model for non-autoregressive mel-spectrogram generation.Source: §4 · p. 4
The authors propose extending Diff-TTS to multi-speaker and emotional TTS in future work.Source: §4 · p. 4

Transformer-based In-Context Score Operator Learning for Diffusion Models2026