Source-linked AI summary
Guided-TTS: A Diffusion Model for Text-to-Speech via Classifier Guidance
Heeseung Kim, Sungwon Kim, Sungroh Yoon
TL;DR
TTS models generally need target-speaker transcripts, leaving long-form untranscribed speech difficult to use for training. Guided-TTS trains an unconditional diffusion model on untranscribed speech and guides synthesis with a separate phoneme classifier, using norm-based guidance to reduce pronunciation errors. It matches prior TTS models on LJSpeech without its transcripts and generalizes to diverse untranscribed datasets, including long-form unsegmented speech.
Problem
Most TTS models require target-speaker transcripts, while using long-form untranscribed speech is challenging because it must be segmented and accurately transcribed.
Source: §1 · p. 1Method
Guided-TTS trains an unconditional diffusion model on untranscribed speech and uses a separately trained phoneme classifier to guide text-conditioned synthesis, with norm-based scaling of the guidance.
Source: §1 · p. 1, §3 · p. 3, §3.2.1 · p. 5Results
Takeaways & Limitations
Abstract
from arXiv · showhide
We propose Guided-TTS, a high-quality text-to-speech (TTS) model that does not require any transcript of target speaker using classifier guidance. Guided-TTS combines an unconditional diffusion probabilistic model with a separately trained phoneme classifier for classifier guidance. Our unconditional diffusion model learns to generate speech without any context from untranscribed speech data. For TTS synthesis, we guide the generative process of the diffusion model with a phoneme classifier trained on a large-scale speech recognition dataset. We present a norm-based scaling method that reduces the pronunciation errors of classifier guidance in Guided-TTS. We show that Guided-TTS achieves a performance comparable to that of the state-of-the-art TTS model, Grad-TTS, without any transcript for LJSpeech. We further demonstrate that Guided-TTS performs well on diverse datasets including a long-form untranscribed dataset.
1. Introduction
Guided-TTS uses an unconditional diffusion model trained on untranscribed speech and a separately trained phoneme classifier to synthesize speech from text without target-speaker transcripts. It matches publicly available high-quality TTS models on LJSpeech and generalizes to diverse untranscribed datasets, including long-form unsegmented speech.
1. Introduction
- Existing TTS models generally require target-speaker transcripts, making long-form untranscribed speech difficult to use because it must be segmented and accurately transcribed.Prior adaptation approaches also rely on challenging-to-train multispeaker models and underperform high-quality single-speaker systems.Source: §1 · p. 1
- An unconditional DDPM trained on untranscribed speech generates mel-spectrograms, while a phoneme classifier trained on large-scale speech-recognition data guides text-conditioned synthesis.Because the untranscribed data need not align with text, the model can train on random speech chunks.Source: §1 · pp. 1–2
- Norm-based guidance balances the classifier gradient against the unconditional score during sampling to reduce mispronunciation caused by guidance errors.The authors identify mispronunciation as a serious consequence of guidance error.Source: §1 · p. 2
- Guided-TTS matches publicly available high-quality TTS models on LJSpeech without LJSpeech transcripts and generalizes to diverse untranscribed datasets, including long-form unsegmented speech.Norm-based guidance significantly reduces pronunciation errors, yielding pronunciation accuracy similar to existing conditional TTS models.Source: §1 · p. 2
2. Background
The background introduces diffusion models as a way to generate data by reversing a noise-corruption process, with a learned score guiding sampling. It then explains classifier guidance as a way to condition generation, with guidance strength trading diversity for fidelity.
2.1. Denoising Diffusion Probablistic Models (DDPM) and Its Variant
- DDPM corrupts data toward random noise in a forward process, then learns the reverse process to generate data from noise.The continuous SDE formulation used here follows Grad-TTS; its reverse process uses the score, the gradient of the log density, to sample data from noise.Source: §2.1 · p. 2
- The score model is trained on noisy samples derived from clean data, using an L2 loss to estimate the score across timesteps.Grad-TTS generates samples by solving the reverse SDE with T = 1 and a fixed discretization strategy.Source: §2.1 · p. 2
2.2. Classifier Guidance
- Classifier guidance conditions an unconditional DDPM by training a classifier to predict the desired condition from noisy samples and using its gradient during sampling.The conditional score replaces the unconditional score when generating samples for a specified condition.Source: §2.2 · p. 2
- Increasing classifier-gradient scale can produce higher-fidelity but less-diverse samples, a trade-off demonstrated for class-conditional image generation.Dhariwal & Nichol (2021) apply this scaling to unconditional and conditional DDPMs.Source: §2.2 · p. 3
3. Guided-TTS
Guided-TTS models speech unconditionally from untranscribed data, then uses classifier guidance to synthesize speech from text without target-speaker transcripts. Its norm-based guidance scales the classifier gradient relative to the unconditional score.
3. Guided-TTS
- Guided-TTS combines an unconditional DDPM with a phoneme classifier, duration predictor, and speaker encoder to generate speech from text without target-speaker transcripts.The paper presents this as an unconditional generative model guided for text-to-speech, rather than a model that directly learns speech generation from text.Source: §3 · p. 3
3.1. Unconditional DDPM
- The unconditional DDPM learns the target speaker’s speech distribution from random chunks of untranscribed audio, avoiding transcript alignment and sentence-level segmentation.It estimates the unconditional score at each timestep with a neural network; mel-spectrograms are treated as single-channel 2D images using a U-Net.Source: §3.1 · p. 3
3.2. Text-to-Speech via Classifier Guidance
- A duration predictor expands text into frame-level phoneme labels, and the phoneme classifier’s gradient is added to the unconditional score to guide sampling.The guidance modules are speaker-dependent; the classifier is trained on speech-recognition data and recognizes frame-level phonemes from mel-spectrograms.Source: §3.2 · pp. 3–4
3.2.1. NORM-BASED GUIDANCE
- Norm-based guidance scales the classifier-gradient norm relative to the unconditional score, addressing the score’s sharp increase near clean data that otherwise weakens guidance and yields mispronunciations.The method also uses a temperature parameter; the authors report that tuning it above 1 helps generate high-quality mel-spectrograms.Source: §3.2.1 · pp. 4–5
4. Experiments
The experiments use LibriSpeech to train speaker-dependent guidance modules and evaluate Guided-TTS on several target-speaker datasets, including unsegmented long-form audio. They compare synthesis settings with and without target-speaker transcripts and describe a CER-based assessment of pronunciation.
Datasets
- The phoneme classifier and duration predictor are trained on LibriSpeech, while the speaker encoder is trained on VoxCeleb2.LibriSpeech has approximately 982 hours from 2,484 speakers; VoxCeleb2 contains more than 1 million utterances from 6,112 speakers.Source: §4 · p. 5
- Evaluation spans LJSpeech, three selected Hi-Fi TTS speakers, and unsegmented Blizzard 2013 audiobook data clipped into five-second chunks.LJSpeech is used for comparisons with transcript-trained baselines; the other target-speaker settings use only untranscribed speech.Source: §4 · p. 5
Evaluation
- Guided-TTS is compared with Glow-TTS and Grad-TTS using pretrained models, and with Grad-TTS trained on ASR-transcribed target data when target transcripts are unavailable.Guided-TTS uses τ = 1.5 and 50 reverse steps; its classifier gradient is withheld initially, then increased linearly to 0.3.Source: §4 · p. 5
- Pronunciation is assessed with character error rate (CER), averaging results across five generated samples per test sentence using a conformer ASR model.The conformer was pretrained on 7,000 hours of speech.Source: §4 · p. 5
5. Results
Guided-TTS is comparable to high-quality TTS models on LJSpeech without LJSpeech transcripts and generalizes to diverse untranscribed datasets, including long-form unsegmented audio. Norm-based guidance improves pronunciation accuracy, which also improves as the phoneme classifier uses more training data.
5.1. Model Comparison
- Guided-TTS achieves performance comparable to Glow-TTS and Grad-TTS on LJSpeech without LJSpeech transcripts, with CER similar to conditional TTS models.Table 1 reports MOS for TTS models on LJSpeech; the supplied passages do not include its numerical values.Source: §5.1 · p. 6, Table 1 · p. 6
5.2. Generalization to Diverse Datasets
- On LJSpeech and Hi-Fi TTS, Guided-TTS is comparable to transcript-trained Grad-TTS on LJSpeech and outperforms Grad-TTS-ASR on Hi-Fi TTS; it also generates high-quality speech from unsegmented Blizzard audio.The authors report low CER across datasets, including Blizzard trained on randomly cropped audio without sentence-level segmentation.Source: §5.2 · p. 6, Table 2 · p. 6
5.3. Analysis
- At an appropriate gradient scale, norm-based guidance produces CER similar to existing TTS models, while classifier guidance produces much worse CER.The authors use a default norm-based gradient scale of s = 0.3, which they report generates high-quality samples matching the given text; scales that are too small or large impair text fidelity or sample quality, respectively.Source: §5.3 · p. 7, Figure 2 · p. 7
- Pronunciation accuracy improves as the phoneme classifier is trained with more LibriSpeech data.The tested training portions were 1% (9 hours), 10% (96 hours), and 100% (960 hours).Source: §5.3 · p. 7, Table 3 · p. 7
6. Related Work
Prior unconditional speech-generation models were used for audio modeling rather than TTS, while existing untranscribed-data approaches relied on pretrained multispeaker TTS models. Guided-TTS applies classifier guidance to unconditional diffusion and uses an ASR dataset to train its phoneme classifier.
Unconditional Speech Generation
- Most unconditional speech models had been used only for audio modeling, and Guided-TTS applies classifier guidance to unconditional generation for TTS.The authors describe this, to their knowledge, as the first use of an unconditional model for TTS with appropriate guidance.Source: §6 · pp. 7–8
Text-to-Speech with Untranscribed Data
- Guided-TTS models unconditional speech from untranscribed target-speaker data and uses a pretrained phoneme classifier to generate samples conditioned on text.The unconditional DDPM and phoneme classifier can be trained on different datasets.Source: §6 · p. 8
- Existing untranscribed-data approaches rely on pretrained multispeaker TTS models, which are difficult to train and can underperform single-speaker models trained on paired speech and text.Guided-TTS instead uses an ASR dataset to train a phoneme classifier and reports performance comparable to other TTS models with untranscribed target-speaker data.Source: §6 · p. 8
- Guided-TTS assumes a large-scale speech-recognition dataset is available, unlike an approach focused on low-resource languages that extracts unsupervised linguistic units.Source: §6 · p. 8
7. Conclusion
Guided-TTS guides an unconditional diffusion speech model with a phoneme classifier trained on LibriSpeech. The authors report performance matching previous TTS models on LJSpeech without transcripts and generalization to diverse untranscribed datasets.
7. Conclusion
- Guided-TTS generates speech from transcripts by guiding an unconditional diffusion-based speech model with a phoneme classifier trained on LibriSpeech.Source: §7 · p. 8
- Guided-TTS matches previous TTS model performance on LJSpeech without transcripts and generalizes to diverse untranscribed datasets using a single phoneme classifier.Source: §7 · p. 9
- The authors believe Guided-TTS can reduce the burden of constructing training datasets for high-quality TTS.Source: §7 · p. 9
A.1. Training Details and Hyperparamters
The unconditional DDPMs are trained on untranscribed target-speaker data, while the phoneme classifier and duration predictor use LibriSpeech with alignment labels. The appendix specifies their architectures, training schedules, and checkpoint selection.
A.1. Training Details and Hyperparamters
- Unconditional DDPMs use untranscribed LJSpeech, Hi-Fi TTS, or Blizzard data; the phoneme classifier and duration predictor are trained on aligned LibriSpeech.Montreal Forced Aligner extracts the alignment labels.Source: §A.1 · p. 13
- The phoneme classifier uses a WaveNet-like architecture with 256 residual channels and six stacks of three dilated-convolution layers.It is trained for 200 epochs with batch size 64; the unconditional DDPMs use batch size 16.Source: §A.1 · p. 13
- Sampling uses the last unconditional-DDPM and speaker-encoder checkpoints, but selects classifier and duration-predictor checkpoints by their respective validation metrics.The classifier uses validation accuracy and the duration predictor uses validation loss.Source: §A.1 · p. 13
A.2. Hardware and Sampling Speed
Guided-TTS runs in real time on an NVIDIA RTX A40 at the evaluation setting of N = 50 reverse steps.
A.2. Hardware and Sampling Speed
- At N = 50 reverse steps, Guided-TTS achieves an RTF of 0.486 on an NVIDIA RTX A40 with 48GB memory.The unconditional score takes 0.184 RTF and classifier-gradient calculation takes 0.291 RTF.Source: §A.2 · p. 13
- Classifier-gradient calculation accounts for 0.291 of the 0.486 RTF, compared with 0.184 for the unconditional score.The reported timings are for evaluation on an NVIDIA RTX A40 with 48GB memory.Source: §A.2 · p. 13
- The reported real-time result is for GPU synthesis using 50 reverse steps; fast inference is not a main objective of Guided-TTS.The experiments use an NVIDIA RTX A40 with 48GB memory.Source: §A.2 · p. 13
A.3. Out-of-Distribution (OoD) Text Robustness
Guided-TTS produces high-quality speech for out-of-distribution text, while norm-based guidance addresses a large imbalance between the unconditional score and classifier gradient. Further experiments report comparable performance with transcribed target-speaker data and show that unconditional DDPM inpainting captures mel-spectrogram dependencies.
A.3. Out-of-Distribution (OoD) Text Robustness
- Guided-TTS produces high-quality speech that faithfully reflects OoD text, whereas Grad-TTS-ASR produces inaccurate, worse-quality samples.The comparison uses 50 randomly selected LJSpeech test sentences; the reported CER indicates text fidelity.Source: §A.3 · p. 13, Table 4 · p. 13
A.4. Norm of the Unconditional Score and Classifier Gradient
- Near t = 0, the unconditional score norm is about 70 times the classifier-gradient norm, reducing guidance’s effect and motivating norm-based scaling.The proposed scaling helps prevent classifier guidance from being ignored and speech from failing to match the text.Source: §A.4 · p. 13
A.5. Guided-TTS with Transribed Speech Data
- Guided-TTS-T uses the target speaker’s transcripts to train its phoneme classifier and duration predictor, unlike Guided-TTS’s LibriSpeech-trained phoneme classifier.Its unconditional DDPM is trained on the target speaker’s untranscribed speech, as in Guided-TTS.Source: §A.5 · p. 14
- With transcribed LJSpeech data, Guided-TTS-T achieves performance similar to Glow-TTS and Grad-TTS using the same amount of training data.Guided-TTS-T trains its phoneme classifier and duration predictor on the target speaker’s dataset, so it does not need a speaker encoder.Source: §A.5 · p. 14
A.6. Inpainting
- Unconditional DDPM inpainting fills masked mel-spectrogram regions and demonstrates learned adjacent frequency and temporal dependencies.Figure 5 shows the original, masked, and inpainted spectrograms for samples from multiple speakers.Source: §A.6 · p. 15, Figure 5 · p. 15