Source-linked AI summary
SC-GlowTTS: an Efficient Zero-Shot Multi-Speaker Text-To-Speech Model
Edresson Casanova, Christopher Shulby, Eren Gölge, Nicolas Michael Müller, Frederico Santos de Oliveira, Arnaldo Candido Junior, Anderson da Silva Soares, Sandra Maria Aluisio, Moacir Antonelli Ponti
TL;DR
Zero-shot multi-speaker TTS must preserve similarity and quality for speakers unseen during training. SC-GlowTTS uses speaker-conditioned GlowTTS with flow-based decoding, three text-encoder variants, and HiFi-GAN adaptation. The system achieves strong unseen-speaker similarity and quality with 11 training speakers, and all models operate in real time with the HiFi-GAN vocoder.
Problem
Zero-shot multi-speaker TTS faces a similarity and quality gap between observed and unobserved speakers, while existing approaches rely heavily on Tacotron 2.
Source: §1 · p. 1Method
SC-GlowTTS combines GlowTTS spectrogram prediction, external speaker embeddings, flow-based decoding, three text encoders, and an adapted HiFi-GAN vocoder.
Source: §1 · p. 1Results
With HiFi-GAN fine-tuning, SECS increased from 0.7641 to 0.8046 for SC-GlowTTS-Trans, while all models were real time on CPU and GPU.
Source: §4 · p. 3Takeaways & Limitations
Abstract
from arXiv · showhide
In this paper, we propose SC-GlowTTS: an efficient zero-shot multi-speaker text-to-speech model that improves similarity for speakers unseen during training. We propose a speaker-conditional architecture that explores a flow-based decoder that works in a zero-shot scenario. As text encoders, we explore a dilated residual convolutional-based encoder, gated convolutional-based encoder, and transformer-based encoder. Additionally, we have shown that adjusting a GAN-based vocoder for the spectrograms predicted by the TTS model on the training dataset can significantly improve the similarity and speech quality for new speakers. Our model converges using only 11 speakers, reaching state-of-the-art results for similarity with new speakers, as well as high speech quality.
1. Introduction
Zero-shot multi-speaker TTS targets voices unseen during training, but similarity and quality remain difficult to match. SC-GlowTTS addresses this with a speaker-conditional GlowTTS architecture, achieving state-of-the-art results with 11 training speakers and improving vocoder-based quality and similarity.
1. Introduction
- Zero-shot TTS addresses interest in synthesizing speakers unseen during training from only a few seconds of speech samples.This setting is distinguished from traditional systems tailored to a single speaker.Source: §1 · p. 1
- Prior work explored speaker embeddings, fine-grained reference encoders, and multiple reference samples to improve similarity for unseen speakers.Reported approaches include Deep Voice 3, Tacotron 2, LDE embeddings, gender-dependent models, and Attentron.Source: §1 · p. 1
- Flow-based methods offer an alternative to the Tacotron 2 focus of existing approaches, with GlowTTS reported as faster while retaining similar quality.GlowTTS is described as allowing speech velocity manipulation and increasing speed by 15.7 times relative to Tacotron 2.Source: §1 · p. 1
- SC-GlowTTS proposes zero-shot multi-speaker TTS using a speaker-conditional architecture for unseen speakers.The model builds on GlowTTS, uses an external speaker encoder, and adapts HiFi-GAN to synthesize waveforms.Source: §1 · p. 1
- 11 speakers in training are sufficient for the proposed approach to achieve state-of-the-art zero-shot multi-speaker TTS results.The paper presents this as a central contribution for synthesizing unseen speakers.Source: §1 · p. 1
- Adjusting a GAN-based vocoder for TTS-predicted spectrograms significantly improves similarity and speech quality for new speakers.The adjustment is performed using spectrograms generated from the TTS model on training data.Source: §1 · p. 1
2. Speaker Conditional GlowTTS Model
SC-GlowTTS extends GlowTTS with speaker conditioning throughout a flow-based decoder and duration predictor, while comparing transformer and two convolutional text encoders. Its inference pipeline predicts an aligned prior, samples a latent variable, and synthesizes mel spectrograms in parallel.
2. Speaker Conditional GlowTTS Model
- The model explores transformer, residual dilated convolutional, and gated convolutional text encoders.The residual encoder uses Mish activations, while the gated encoder uses nine convolutional blocks with dropout, convolution, and layer normalization.Source: §2 · pp. 1–2
- The architecture is presented as Speaker Conditional GlowTTS General Architecture in Figure 1.The figure illustrates the model during inference.Source: Figure 1 · p. 2, §2 · p. 2
- SC-GlowTTS conditions all 12 affine coupling blocks and the duration predictor on external speaker embeddings for zero-shot synthesis.The flow-based decoder retains GlowTTS’s configuration while incorporating speaker information into its transformations and duration prediction.Source: §2 · p. 2
- During inference, the text encoder and duration predictor provide the prior distribution and alignment, after which a latent variable is transformed into mel spectrograms by the inverted flow decoder.Monotonic Alignment Search is used during training but not inference.Source: §2 · p. 2
- The transformer, residual convolutional, and gated convolutional variants are named SC-GlowTTS-Trans, SC-GlowTTS-Res, and SC-GlowTTS-Gated.These names distinguish the three text-encoder configurations.Source: §2 · p. 2
3. Experiments
The experiments train and compare Tacotron 2 with three SC-GlowTTS encoder variants on VCTK, using unseen test speakers and a separately adapted HiFi-GAN vocoder. Evaluation combines speaker similarity, perceptual quality, and real-time synthesis speed.
3.1. Speaker Encoder
- The speaker encoder uses three LSTM layers and a linear output layer to produce speaker representations.It is trained with Angular Prototypical loss using audio from approximately 25k speakers across several datasets.Source: §3.1 · p. 2, §3.3 · p. 2
3.2. Zero-Shot Multi-Speaker Tacotron 2
- The Tacotron 2 baseline concatenates speaker embeddings with attention inputs and uses Double Decoder Consistency, guided attention, and gradual reduction-rate training.The coarse decoder uses R = 7, while the fine decoder decreases from R = 7 to R = 5, R = 3, and R = 2 during training.Source: §3.2 · p. 2
3.3. Audio datasets
- VCTK supplies 44 hours from 109 speakers, with 11 speakers held out for testing across seven female and four male voices.The held-out speakers are absent from both training and validation, and represent seven accents.Source: §3.3 · p. 2
3.4. Experimental setup
- Four experiments compare Tacotron 2 with SC-GlowTTS-Trans, SC-GlowTTS-Res, and SC-GlowTTS-Gated.The SC-GlowTTS variants are each trained for 150k steps, while Tacotron 2 is trained for 210k steps.Source: §3.4 · p. 2
- All experiments use phoneme inputs, VCTK train-validation-test partitions, RAdam optimization, and validation loss to select checkpoints.GlowTTS-based models additionally insert blank tokens between phonemes.Source: §3.4 · p. 3
- HiFi-GAN is first trained on LibriTTS and then fine-tuned for 190k steps using spectrograms synthesized from the TTS models on VCTK training and validation data.Teacher forcing preserves Tacotron alignments, while MAS aligns SC-GlowTTS outputs with encoder outputs.Source: §3.4 · p. 3
4. Results and Discussion
The experiments evaluate synthesis speed, speech quality, and speaker similarity across SC-GlowTTS variants, Tacotron 2, and Attentron. HiFi-GAN fine-tuning improves unseen-speaker similarity and quality, while SC-GlowTTS remains real-time and competitive with prior systems.
Evaluation
- The evaluation used MOS, Sim-MOS, and SECS, with SECS computed from speaker-encoder cosine similarity and higher values indicating stronger similarity.The study used 15 professional collaborators per audio for MOS and included 11 unseen test speakers.Source: §4 · p. 3
- The experimental reference for each test speaker was the fifth VCTK sentence, selected because every speaker uttered the same 20-word sentence.Using the same content and length standardizes the reference samples used for speaker-embedding extraction.Source: §4 · p. 3
Runtime
- SC-GlowTTS-Gated was fastest on CPU and GPU, followed by SC-GlowTTS-Res, while SC-GlowTTS-Trans remained faster than Tacotron 2.After HiFi-GAN integration, all models were real-time on both CPU and GPU.Source: §4 · p. 3, Table 1 · p. 4
Speaker similarity
- SC-GlowTTS-Trans achieved the best SECS with HiFi-GAN-FT, reaching 0.8046, while fine-tuning increased SECS for every evaluated model.SECS increased from 0.7589 to 0.7791 for Tacotron 2, 0.7641 to 0.8046 for SC-GlowTTS-Trans, 0.7440 to 0.7969 for SC-GlowTTS-Res, and 0.7432 to 0.7849 for SC-GlowTTS-Gated.Source: §4 · p. 3
- The best model reached Sim-MOS 3.999 ± 0.07 and exceeded Attentron's zero-shot and few-shot results, which reached 3.30 ± 0.06 and 3.57 ± 0.05.Attentron's few-shot result used eight reference samples, whereas the comparison model used the zero-shot setting.Source: §4 · pp. 3–4
Speech quality
- HiFi-GAN-FT increased MOS for new speakers from 3.57 to 3.74, 3.65 to 3.78, 3.45 to 3.70, and 3.55 to 3.82 across the four evaluated models.The SC-GlowTTS-Gated model with HiFi-GAN-FT was closest to the ground-truth MOS of 4.12, reaching 3.82.Source: §4 · p. 4
5. SC-GlowTTS performance with few speakers
SC-GlowTTS-Trans was fine-tuned with only 11 training speakers and achieved performance comparable to Tacotron 2 trained with 98 speakers. The authors report this as a 9.8-times smaller dataset setting.
Few-speaker setup
- The few-speaker training set contained 11 speakers, with seven female and four male speakers, selected to represent accents.The authors used a held-out validation subset from these speakers and the same test set as the main experiments.Source: §5 · p. 4
Training procedure
- The model was pretrained on LJSpeech, fine-tuned on the 11-speaker set, and paired with a HiFi-GAN vocoder adjusted using the same technique.SC-GlowTTS-Trans was pretrained for 290k steps, fine-tuned for 70k steps, and the vocoder was adjusted for 95k steps.Source: §5 · p. 4
Few-speaker performance
- SC-GlowTTS-Trans achieved comparable performance to Tacotron 2 while using a 9.8-times smaller dataset.The 11-speaker model reached SECS 0.7707, MOS 3.71 ± 0.07, and Sim-MOS 3.93 ± 0.08, compared with Tacotron 2's SECS 0.7791, MOS 3.74, and Sim-MOS 3.951 ± 0.07 using 98 speakers.Source: §5 · p. 4
6. Zero-Shot Voice Conversion
SC-GlowTTS conditions its flow-based decoder on external speaker embeddings to support zero-shot voice conversion. The encoder remains independent of speaker identity, allowing voice conversion through the decoder.
Zero-Shot Voice Conversion
- External speaker embeddings condition all decoder coupling layers so SC-GlowTTS can resemble speakers unseen during training.The model uses the decoder for zero-shot voice conversion while keeping speaker identity out of the encoder distribution.Source: §6 · p. 4
7. Conclusions and future work
SC-GlowTTS achieves state-of-the-art ZS-TTS results, with the transformer encoder providing the best unseen-speaker similarity. HiFi-GAN adjustment improves similarity and MOS, and the authors identify few-shot extension as future work.
Conclusions
- The transformer-based encoder gave the best similarity for speakers unseen during training, and SC-GlowTTS models outperformed Tacotron 2.The authors report state-of-the-art ZS-TTS results and successful operation with only 11 training speakers.Source: §7 · p. 4
- HiFi-GAN adjustment improved both similarity and synthesized-speech quality for unseen speakers.The adjustment was applied to spectrograms predicted by the TTS model on training and validation data.Source: §7 · p. 4