Source-linked AI summary
AdaSpeech 4: Adaptive Text to Speech in Zero-Shot Scenarios
Yihan Wu, Xu Tan, Bohan Li, Lei He, Sheng Zhao, Ruihua Song, Tao Qin, Tie-Yan Liu
TL;DR
Zero-shot TTS must generalize to diverse new speakers, while few-shot adaptation requires target-speaker data and fine-tuning. AdaSpeech 4 combines basis-vector speaker representations, conditional layer normalization, and distribution-based supervision; it achieves better voice quality and similarity than baselines across multiple datasets.
Problem
Zero-shot TTS must generalize across diverse unseen speaker characteristics, while few-shot adaptation requires target-speaker data and source-model fine-tuning.
Source: §1 · p. 1Method
AdaSpeech 4 extracts speaker representations through attention-weighted basis vectors, integrates them with conditional layer normalization, and uses distribution-based supervision for generated mel-spectrograms.
Source: §1 · pp. 1–2Results
Across LibriTTS, VCTK, and LJSpeech, AdaSpeech 4 achieves better voice quality and similarity in MOS and SMOS than baseline methods in zero-shot scenarios.
Source: §1 · p. 2Takeaways & Limitations
AdaSpeech 4 targets data- and computation-efficient synthesis for new speakers by modeling them from reference speech without adapting the source TTS model.
Source: §1 · p. 1
Abstract
from arXiv · showhide
Adaptive text to speech (TTS) can synthesize new voices in zero-shot scenarios efficiently, by using a well-trained source TTS model without adapting it on the speech data of new speakers. Considering seen and unseen speakers have diverse characteristics, zero-shot adaptive TTS requires strong generalization ability on speaker characteristics, which brings modeling challenges. In this paper, we develop AdaSpeech 4, a zero-shot adaptive TTS system for high-quality speech synthesis. We model the speaker characteristics systematically to improve the generalization on new speakers. Generally, the modeling of speaker characteristics can be categorized into three steps: extracting speaker representation, taking this speaker representation as condition, and synthesizing speech/mel-spectrogram given this speaker representation. Accordingly, we improve the modeling in three steps: 1) To extract speaker representation with better generalization, we factorize the speaker characteristics into basis vectors and extract speaker representation by weighted combining of these basis vectors through attention. 2) We leverage conditional layer normalization to integrate the extracted speaker representation to TTS model. 3) We propose a novel supervision loss based on the distribution of basis vectors to maintain the corresponding speaker characteristics in generated mel-spectrograms. Without any fine-tuning, AdaSpeech 4 achieves better voice quality and similarity than baselines in multiple datasets.
1. Introduction
Zero-shot adaptive TTS avoids target-speaker fine-tuning, but must generalize across diverse speaker characteristics. AdaSpeech 4 addresses this challenge through changes to speaker representation, conditioning, and supervision.
1. Introduction
- Zero-shot adaptive TTS uses reference speech without adapting the source model, but diverse unseen speakers demand strong generalization; few-shot adaptation instead requires target data and fine-tuning.Fine-tuning also incurs computation cost when serving many new speakers.Source: §1 · p. 1
- AdaSpeech 4 represents speaker characteristics with basis vectors and extracts a speaker representation by attention-weighted combination to improve generalization to new speakers.The basis vectors are initialized from speaker-embedding cluster centers and encouraged to be dissimilar with a regularization loss.Source: §1 · pp. 1–2
- Conditional layer normalization integrates the extracted speaker representation into the phoneme encoder and mel-spectrogram decoder, improving generalization over directly using the representation as decoder input.Its scale and bias vectors are generated by model parameters conditioned on the extracted representation.Source: §1 · p. 2
- A novel loss based on the basis-vector distribution supervises generated mel-spectrograms to preserve speaker representation and resemble the reference speech.The paper presents this as a way to improve similarity in zero-shot scenarios.Source: §1 · p. 2
- Across LibriTTS, VCTK, and LJSpeech, AdaSpeech 4 achieves better voice quality and similarity in MOS and SMOS than baseline methods in zero-shot scenarios.Ablation studies and method analyses also verify the effectiveness of each design.Source: §1 · p. 2
2. Proposed Method
The method addresses three stages of zero-shot speaker modeling: extracting a representation, conditioning the TTS model on it, and supervising generated mel-spectrograms. It uses attention over basis vectors, conditional layer normalization, and a distribution-based loss.
2.1. Extracting Speaker Representation by Basis Vectors
- Speaker representations are extracted by attention-weighting shared basis vectors, initialized from speaker-embedding cluster centers and regularized to remain dissimilar.The attention mechanism forms new combination weights over basis vectors learned from training speakers.Source: §2.1 · p. 2
2.2. Integrating Speaker Representation with Conditional Layer Normalization
- Conditional layer normalization derives adaptive scale and bias from the extracted representation and is applied in both the phoneme encoder and mel decoder.It replaces conventional normalization in Transformer self-attention and feed-forward networks.Source: §2.2 · p. 3
2.3. Supervising Speaker Representation in Synthesized Speech with Distribution Loss
- A KL-divergence supervision loss aligns reference and generated speech attention-weight distributions over shared basis vectors, making speaker characteristics more controllable in synthesized speech.The loss compares attention weights computed for reference and generated speech.Source: §2.3 · p. 3
3. Experiments and Results
AdaSpeech 4 improves zero-shot voice quality and similarity over baselines across LibriTTS, VCTK, and LJSpeech. Ablations support the roles of its speaker-extraction, conditioning, and supervision components.
3.1. Datasets and Experiments Settings
- Experiments train on LibriTTS and evaluate zero-shot synthesis on unseen LibriTTS speakers, VCTK, and LJSpeech; each evaluation speaker contributes one reference audio.The study samples ten speakers from multi-speaker datasets and one from LJSpeech, then synthesizes 15 sentences per speaker for human evaluation.Source: §3.1 · pp. 3–4
3.2. Similarity and Quality Comparison with Baselines
- AdaSpeech 4 improves SMOS across LibriTTS, VCTK, and LJSpeech while maintaining good or slightly improving MOS versus baseline methods.Listening tests used 20 native judges per sentence; LibriTTS and VCTK scores were averaged across multiple speakers.Source: §3.2 · p. 4
3.3. Ablation Studies
- Removing basis-vector initialization, regularization, or the speaker basis vectors reduces voice quality or similarity; regularization removal causes the largest CMOS drop, −0.46.Removing k-means initialization causes a −0.36 CMOS voice-quality drop.Source: §3.3 · p. 4, Table 2 · p. 4
- Discarding conditional layer normalization in the phoneme encoder, or in both the phoneme encoder and mel decoder, impairs voice quality and speaker similarity.Source: §3.3 · p. 4
- Removing distribution loss lowers SMOS, while embedding-level cosine supervision alone or jointly with it brings no obvious gain.The cosine loss compares reference and generated speaker embeddings.Source: §3.3 · p. 4
- Voice quality and similarity decline as the basis-vector count falls below 2,000, with no obvious gain above 2,000.The experiments therefore use 2,000 basis vectors.Source: §3.3 · p. 4, Figure 2 · p. 4
4. Conclusion
AdaSpeech 4 systematically models speaker characteristics to improve generalization in zero-shot TTS. Its experiments demonstrate high-quality, similar speech synthesis, while future work will evaluate more diverse speaker characteristics and improve prosody and expressiveness.
- AdaSpeech 4 extracts speaker representations with basis vectors, integrates them through conditional layer normalization, and uses distribution-level supervision to maintain speaker representation in generated speech.Source: §4 · p. 4