Source-linked AI summary
AdaSpeech: Adaptive Text to Speech for Custom Voice
Mingjian Chen, Xu Tan, Bohan Li, Yanqing Liu, Tao Qin, Sheng Zhao, Tie-Yan Liu
TL;DR
Custom voice must adapt TTS models from few recordings while handling acoustic differences from source data and limiting per-speaker memory. AdaSpeech models acoustic conditions at multiple granularities and uses conditional layer normalization for efficient adaptation; experiments report better adaptation quality than baselines with about 5K speaker-specific parameters.
Problem
Custom voice adaptation must handle diverse acoustic conditions and keep per-speaker adaptation parameters small while maintaining high voice quality.
Method
AdaSpeech combines multi-granularity acoustic condition modeling with conditional layer normalization in a pre-training, fine-tuning, and inference pipeline.
Results
AdaSpeech achieves better adaptation quality than baseline methods with only about 5K specific parameters for each speaker.
Takeaways & Limitations
AdaSpeech supports custom voice across different acoustic conditions with few speaker-specific parameters and high voice quality.
Takeaways & Limitations
The approach does not consider language-level acoustic conditions in a multilingual setting and leaves that setting for future work.
Abstract
from arXiv · showhide
Custom voice, a specific text to speech (TTS) service in commercial speech platforms, aims to adapt a source TTS model to synthesize personal voice for a target speaker using few speech data. Custom voice presents two unique challenges for TTS adaptation: 1) to support diverse customers, the adaptation model needs to handle diverse acoustic conditions that could be very different from source speech data, and 2) to support a large number of customers, the adaptation parameters need to be small enough for each target speaker to reduce memory usage while maintaining high voice quality. In this work, we propose AdaSpeech, an adaptive TTS system for high-quality and efficient customization of new voices. We design several techniques in AdaSpeech to address the two challenges in custom voice: 1) To handle different acoustic conditions, we use two acoustic encoders to extract an utterance-level vector and a sequence of phoneme-level vectors from the target speech during training; in inference, we extract the utterance-level vector from a reference speech and use an acoustic predictor to predict the phoneme-level vectors. 2) To better trade off the adaptation parameters and voice quality, we introduce conditional layer normalization in the mel-spectrogram decoder of AdaSpeech, and fine-tune this part in addition to speaker embedding for adaptation. We pre-train the source TTS model on LibriTTS datasets and fine-tune it on VCTK and LJSpeech datasets (with different acoustic conditions from LibriTTS) with few adaptation data, e.g., 20 sentences, about 1 minute speech. Experiment results show that AdaSpeech achieves much better adaptation quality than baseline methods, with only about 5K specific parameters for each speaker, which demonstrates its effectiveness for custom voice. Audio samples are available at https://speechresearch.github.io/adaspeech/.
1 INTRODUCTION
Custom voice adapts TTS models to personal voices from few recordings, but differing acoustic conditions and the memory–quality trade-off make adaptation difficult. AdaSpeech addresses these challenges through acoustic condition modeling and conditional layer normalization, achieving improved adaptation quality with about 5K speaker-specific parameters.
- Motivation: Few adaptation data can reduce the naturalness and similarity of adapted voices, especially when users’ recordings differ acoustically from source training data.Differences include prosody, style, emotion, accent, and recording environment; more adaptation parameters can improve quality but increase memory and serving costs.
- Method: AdaSpeech uses utterance-level and phoneme-level acoustic encoders during training to represent global and local conditions, then predicts phoneme-level conditions during inference.Inference extracts an utterance-level vector from reference speech and uses an acoustic predictor built on the phoneme encoder.
- Method: Conditional layer normalization lets AdaSpeech fine-tune decoder parameters conditioned on speaker embeddings instead of adapting the whole model.The design reduces adaptation parameters and memory storage while retaining high-quality adaptation voice.
- Evaluation: AdaSpeech is evaluated by pre-training on LibriTTS and adapting to VCTK and LJSpeech with different adaptation settings.The reported evaluation uses MOS and SMOS to assess adaptation voice quality.
- Results: About 5K specific parameters per speaker demonstrate AdaSpeech’s effectiveness for custom voice.The paper reports better adaptation quality than baseline methods while using this small speaker-specific parameter set.
2 ADASPEECH
AdaSpeech extends a FastSpeech 2 backbone with multi-granularity acoustic-condition modeling and conditional layer normalization for efficient custom-voice adaptation. Its pipeline pre-trains shared components, fine-tunes speaker-specific parameters, and deploys compact voice parameters with the shared model.
- Overall Design: AdaSpeech uses a FastSpeech 2 backbone comprising a phoneme encoder, mel-spectrogram decoder, and variance adaptor.The variance adaptor supplies duration, pitch, and energy information to the phoneme hidden sequence.
- Acoustic Condition Modeling: Acoustic conditions are modeled at speaker, utterance, and phoneme levels to represent increasingly fine-grained characteristics of speech.The design addresses acoustic variation such as speaker characteristics, prosody, accents, pitches, and temporal environmental noise.
- Acoustic Condition Modeling: Two acoustic encoders extract utterance-level and phoneme-level information, which are added to the phoneme hidden sequence during training.The utterance encoder produces a pooled vector, while phoneme-level representations are formed by averaging mel frames aligned to each phoneme.
- Acoustic Condition Modeling: During inference, a reference speech provides the utterance-level vector, while a phoneme-level acoustic predictor estimates phoneme-level vectors from the phoneme hidden sequence.This replaces direct extraction of phoneme-level representations from target speech at inference time.
- Acoustic Condition Modeling: The method does not model frame-level acoustic conditions and leaves language-level conditions outside its multilingual scope.The authors state that frame-level conditions have marginal benefits relative to their prediction difficulty and defer multilingual settings.
- Conditional Layer Normalization: Conditional layer normalization uses speaker embeddings to generate decoder scale and bias vectors, and adaptation fine-tunes these conditional parameters together with the speaker embedding.Other model parameters, including the acoustic encoders and phoneme-level predictor, remain fixed during fine-tuning; inference stores calculated speaker-specific vectors with shared parameters.
3 EXPERIMENTAL SETUP
AdaSpeech is evaluated by adapting a LibriTTS-trained source model to speakers in VCTK, LJSpeech, and LibriTTS under few-shot conditions. Human ratings and ablations assess voice quality, acoustic-condition modeling, parameter efficiency, and sensitivity to adaptation-data size.
- Datasets: The source model is trained on LibriTTS and adapted to VCTK and LJSpeech, whose acoustic conditions differ from LibriTTS, as well as to LibriTTS speakers.LibriTTS contains 2456 speakers and 586 hours; VCTK contains 108 speakers and 44 hours, while LJSpeech contains 24 hours from one speaker.
- Adaptation settings: Each selected speaker is adapted with K = 20 randomly chosen sentences, while smaller adaptation sets are also studied.The original test sets corresponding to adaptation speakers are used for evaluation.
- Model and adaptation configuration: AdaSpeech adapts speaker embeddings and conditional layer-normalization parameters, requiring about 4.9K stored parameters per custom voice when h = 256 and C = 9.The corresponding adaptation process has about 1.2M parameters, compared with 31M for the whole model.
- Evaluation: Naturalness and speaker similarity are evaluated with MOS and SMOS, respectively, using 20 judgments per sentence.Scores are compared across ground-truth, vocoder, baseline, and AdaSpeech settings.
- Main results: AdaSpeech improves both MOS and SMOS over speaker-embedding-only adaptation across all three datasets and slightly exceeds decoder fine-tuning with far fewer parameters.Cross-domain adaptation is more difficult than in-domain adaptation, with larger gaps from the ground-truth mel-plus-vocoder setting on LJSpeech and VCTK.
- Ablation and data-size analysis: Removing utterance-level acoustic modeling, phoneme-level acoustic modeling, or conditional layer normalization lowers VCTK CMOS, while reducing adaptation data below 10 sentences causes a rapid MOS decline.The ablation evaluates each component separately; data-size effects are measured on VCTK and LJSpeech.
5 CONCLUSIONS
The paper presents AdaSpeech as an adaptive TTS system for custom voices with varied acoustic conditions and limited per-speaker storage. Its conclusions emphasize acoustic-condition modeling and conditional layer normalization, while identifying broader acoustic coverage and further compression as future work.
- Contributions: AdaSpeech combines acoustic-condition modeling with conditional layer normalization to support custom voices across varied acoustic conditions using few adaptation parameters.The system uses a pre-training, fine-tuning, and inference pipeline.
- Future work: Future work includes modeling noisier and more diverse acoustic conditions, adapting from untranscribed data, and further compressing the model.
- Method analysis: Fine-tuning the decoder’s last linear layer and last feed-forward network can outperform fine-tuning other decoder parts in the preliminary study.