Source-linked AI summary
Zero-Shot Multi-Speaker Text-To-Speech with State-of-the-art Neural Speaker Embeddings
Erica Cooper, Cheng-I Lai, Yusuke Yasuda, Fuming Fang, Xin Wang, Nanxin Chen, Junichi Yamagishi
TL;DR
Zero-shot adaptation of end-to-end TTS to unseen speakers remains a challenge despite good similarity for speakers seen during training. The paper compares neural speaker embeddings in multi-speaker TTS and finds that LDE-based embeddings improve unseen-speaker similarity and naturalness, while also examining their speaker-verification performance.
Problem
Speaker adaptation with embeddings performs well for speakers seen during TTS training, but zero-shot adaptation to unseen speakers remains limited.
Method
The study extends Tacotron into multi-speaker TTS and compares neural speaker embeddings, including LDE-based representations, for unseen-speaker adaptation.
Results
LDE-based neural speaker embeddings improve speaker similarity and naturalness for unseen speakers compared with x-vectors in zero-shot TTS adaptation.
Takeaways & Limitations
LDE-based embeddings can support zero-shot speaker adaptation while narrowing, but not eliminating, the gap between seen- and unseen-speaker similarity.
Takeaways & Limitations
A remaining gap between seen- and unseen-speaker similarity indicates overfitting to seen speakers and leaves room for speaker-space augmentation.
Abstract
from arXiv · showhide
While speaker adaptation for end-to-end speech synthesis using speaker embeddings can produce good speaker similarity for speakers seen during training, there remains a gap for zero-shot adaptation to unseen speakers. We investigate multi-speaker modeling for end-to-end text-to-speech synthesis and study the effects of different types of state-of-the-art neural speaker embeddings on speaker similarity for unseen speakers. Learnable dictionary encoding-based speaker embeddings with angular softmax loss can improve equal error rates over x-vectors in a speaker verification task; these embeddings also improve speaker similarity and naturalness for unseen speakers when used for zero-shot adaptation to new speakers in end-to-end speech synthesis.
1. INTRODUCTION
The paper examines whether state-of-the-art neural speaker embeddings improve zero-shot adaptation to speakers unseen during TTS training. It extends multi-speaker TTS to compare embedding types and relates generated-voice quality to automatic speaker-verification accuracy.
- Adapting end-to-end TTS models to arbitrary new speakers with small amounts of data remains challenging.
- Transfer learning trains speaker-embedding networks separately, enabling robust speaker representations from large, potentially untranscribed datasets.
- Recent speaker-recognition advances include x-vectors, learnable dictionary encoding, and angular softmax for more effective speaker representations.
- The study extends an improved Tacotron system into multi-speaker TTS to test neural embeddings on speakers unseen during TTS training.
- The paper investigates whether the best embeddings for automatic speaker verification are also best for TTS modeling of unseen speakers.
2. NEURAL SPEAKER EMBEDDINGS
The speaker-recognition systems extract utterance-level embeddings by encoding frame-level features, pooling them, and classifying speaker identity. The paper compares conventional statistical pooling with LDE pooling and normal softmax with angular softmax.
- A typical end-to-end speaker-recognition system combines an encoder, statistical pooling layer, and speaker classifier.The pooling layer converts frame-level representations into a fixed-dimensional utterance embedding.
- x-vectors use TDNN plus statistical pooling, whereas LDE embeddings use ResNet34 plus learnable dictionary encoding.
- Pooling methods: Statistical pooling concatenates frame-level means and standard deviations into a fixed-dimensional vector.
- Pooling methods: LDE learns C cluster centers, assigns frames soft weights using distances, and aggregates weighted representations over time into an utterance-level embedding.The aggregation produces concatenated cluster means and standard deviations.
- Classifier: Angular softmax introduces angular margins between speaker classes to produce more separable embedding representations.The study evaluates both normal softmax and angular softmax.
3. MULTI-SPEAKER TTS MODEL ARCHITECTURE
The proposed multi-speaker TTS model conditions Tacotron-based synthesis on speaker embeddings while using self-attention and dual attention to model speech from phoneme inputs. Zero-shot adaptation averages embeddings from untranscribed target-speaker utterances without fine-tuning.
- Speaker embeddings are supplied as additional inputs to condition speaker characteristics in the multi-speaker TTS system.
- The architecture extends Tacotron with encoder self-attention, forward attention, additive attention, and a decoder that outputs 80-dimensional mel-spectrograms.
- The system uses phoneme inputs produced after rule-based text normalization and conversion with flite.
- During zero-shot adaptation, the speaker encoder averages embeddings from untranscribed target-speaker utterances and uses the result to generate mel spectrograms without fine-tuning.
- A WaveNet vocoder converts the predicted mel spectrograms into 16-bit, 16kHz waveforms.
4. EXPERIMENTS
The experiments compare speaker-embedding architectures and TTS input configurations for seen and unseen speakers, using verification, cosine similarity, and listening tests. LDE-based embeddings improve unseen-speaker outcomes over x-vectors, with LDE-3 best on the completely unseen test set.
- 4.1. Speaker Verification: Speaker verification systems use TDNN+SP x-vectors or ResNet34+LDE embeddings trained on VoxCeleb data.The verification evaluation uses the original VoxCeleb1 test set.
- 4.2. Preliminary Experiments for Speaker Similarity in TTS: The preliminary TTS study varies gender-dependent versus gender-independent training and four locations for injecting speaker embeddings.The tested locations include attention, prenet plus attention, and prenet plus attention plus postnet.
- 4.2. Preliminary Experiments for Speaker Similarity in TTS: Gender-dependent training with x-vectors at both the prenet and attention mechanism produced the best unseen-speaker similarity and was selected for later experiments.Cosine similarity compares speaker embeddings extracted from synthesized speech with those from the target speaker’s actual speech.
- 4.3. Comparing Different Embeddings for Speaker Similarity: The main evaluation trains 15 TTS models using 14 LDE variants and x-vectors, then measures naturalness and speaker similarity for seen and unseen speakers.The listening test uses MOS for naturalness and DMOS for speaker similarity, with 463 participants.
- 4.3. Comparing Different Embeddings for Speaker Similarity: LDE-based neural embeddings improve unseen-speaker similarity and naturalness over x-vectors, while LDE-3 is best on the completely unseen test set.LDE-3 significantly outperforms x-vectors for naturalness (p=5.9e-11) and speaker similarity (p=0.02); angular softmax and postprocessing have small impact.
5. CONCLUSIONS
LDE-based neural speaker embeddings improve speaker similarity and naturalness for unseen speakers and support zero-shot adaptation. However, a seen–unseen similarity gap remains, and future work targets overfitting and finer-grained speaker characteristics.
- LDE-based neural speaker embeddings improve speaker similarity and naturalness for unseen speakers in zero-shot adaptation.
- A remaining gap between seen and unseen speaker similarity indicates that the TTS model may still overfit to seen speakers.Future work will explore speaker-space augmentation and evaluate dialect and speaking style.