Source-linked AI summary
AdaDurIAN: Few-shot Adaptation for Neural Text-to-Speech with DurIAN
Zewang Zhang, Qiao Tian, Heng Lu, Ling-Hui Chen, Shan Liu
TL;DR
Few-shot TTS must maintain pronunciation, naturalness, and cross-lingual speaking with very limited monolingual data, despite instability in attention-based systems. AdaDurIAN addresses this with an improved DurIAN average model and shared speaker-independent content encoding, and evaluations report better naturalness, speaker similarity, cross-lingual speaking, and promising emotion transfer than the Tacotron-like baseline.
Problem
Few-shot TTS lacks sufficient balanced recordings for new speakers, while attention-based systems can be unstable and perform poorly in limited-data cross-lingual adaptation.
Source: §1 · p. 1Method
AdaDurIAN trains an improved DurIAN-based average model with a speaker-independent content encoder and adapts new speakers by fixing shared linguistic modules while tuning speaker-specific components.
Source: §1 · p. 1, §2.1 · p. 2, §2.1.1 · p. 2Results
AdaDurIAN reports higher MOS naturalness, more speaker-similarity preferences, and especially fluent cross-lingual speaking than the Tacotron-like model, with promising few-shot emotion transfer.
Source: §3.3.1 · p. 4, §3.3.2 · p. 4, §4 · p. 4Takeaways & Limitations
Abstract
from arXiv · showhide
This paper investigates how to leverage a DurIAN-based average model to enable a new speaker to have both accurate pronunciation and fluent cross-lingual speaking with very limited monolingual data. A weakness of the recently proposed end-to-end text-to-speech (TTS) systems is that robust alignment is hard to achieve, which hinders it to scale well with very limited data. To cope with this issue, we introduce AdaDurIAN by training an improved DurIAN-based average model and leverage it to few-shot learning with the shared speaker-independent content encoder across different speakers. Several few-shot learning tasks in our experiments show AdaDurIAN can outperform the baseline end-to-end system by a large margin. Subjective evaluations also show that AdaDurIAN yields higher mean opinion score (MOS) of naturalness and more preferences of speaker similarity. In addition, we also apply AdaDurIAN to emotion transfer tasks and demonstrate its promising performance.
1. Introduction
AdaDurIAN addresses the difficulty of adapting TTS to new speakers from only a few minutes of monolingual data, especially when robust alignment and cross-lingual pronunciation are required. It combines a DurIAN-based architecture with selective adaptation and reports gains over Tacotron-like systems in naturalness, speaker similarity, and cross-lingual speaking.
Limitations of existing TTS
- Attention-based end-to-end TTS can produce skipping, repetition, long pauses, and attention collapse on abnormal or out-of-domain text.Few-shot adaptation can also suffer from instability and poor cross-lingual speaking when attention models are fine-tuned with limited data.Source: §1 · p. 1
Motivation
- Few-shot TTS is motivated by the impracticality and expense of collecting at least ten hours of balanced, professional recordings for each speaker.The problem is especially acute for speakers such as pop stars, for whom only limited broadcasts or performances may be available.Source: §1 · p. 1
Contribution
- AdaDurIAN uses a DurIAN-based adaptive TTS system to synthesize natural cross-lingual speech in a new speaker’s voice from only a few minutes of monolingual data.Its approach uses a speaker-independent content encoder and selective adaptation rather than relying on unrestricted end-to-end fine-tuning.Source: §1 · p. 1
- AdaDurIAN combines phoneme and tone-or-stress sequences, speaker characteristics in encoder states, selective fine-tuning of speaker embedding and decoder, and a time-delayed LSTM post-net.These design choices target pronunciation accuracy, cross-lingual speaking, and streaming mel-spectrogram generation.Source: §1 · p. 1
Reported outcomes
- Evaluations report higher naturalness, more speaker-similarity preferences, and especially fluent cross-lingual speaking than the Tacotron-like baseline, alongside promising few-shot emotion transfer.The paper’s abstracted results also identify emotion transfer as an additional application of AdaDurIAN.Source: §1 · p. 1
2. The proposed method
AdaDurIAN improves DurIAN for multi-speaker, multi-style, and multilingual synthesis by separating speaker-independent linguistic processing from speaker-dependent generation. Its few-shot strategy fixes shared linguistic modules and optimizes only speaker-specific components, while duration-based alignment supports robust synthesis.
2.1. Architecture of AdaDurIAN
- AdaDurIAN comprises a speaker-independent content encoder, a duration-predicting alignment model, and an autoregressive decoder that generates five mel-spectrogram frames at a time.The architecture extends original single-speaker DurIAN to multi-speaker, multi-style, and multilingual synthesis.Source: §2.1 · p. 2
2.1.1. Speaker-independent Content Encoder
- The content encoder consumes phoneme, tone-or-stress, and prosodic-boundary sequences to learn speaker-independent linguistic transformations across speakers.State skipping is used to produce hidden states containing global linguistic feature transformations.Source: §2.1.1 · p. 2
2.1.2. Alignment Model
- Speaker, emotion, and language embeddings are concatenated into expanded encoder states, which are repeated by predicted durations and combined with relative position encodings.Language code switching depends on the language of the current phoneme, while the alignment model keeps its trainable embeddings separate from the content encoder.Source: §2.1.2 · p. 2
2.1.3. Decoder
- The decoder uses residual LSTM layers and a five-frame time-delayed LSTM post-net instead of CBHG, enabling streaming synthesis and 17-times-faster-than-real-time inference on two CPU cores.The time-delayed post-net is reported to improve predicted mel-spectrogram quality while supporting production deployment.Source: §2.1.3 · p. 2
2.2. Speaker Adaptation Strategy
- Fixing the shared linguistic modules is evaluated with word error rate because scarce, imbalanced speaker data can otherwise leave new-speaker models unable to synthesize out-of-domain words.The reported rationale is that fixed parameters remain in the same distribution space despite limited unbalanced adaptation data.Source: §2.2 · p. 2
- Few-shot adaptation fixes phone, tone-or-stress, language, emotion embeddings, and the encoder in a shuffled multi-speaker average model, then optimizes speaker embedding and decoder components.This transfers shared linguistic knowledge to new speakers while limiting adaptation to components that can be learned from scarce data.Source: §2.2 · p. 2
3. Experiments
The experiments evaluate AdaDurIAN against SMA through pronunciation, few-shot speaker adaptation, and emotion-transfer tasks using limited data. Results report lower pronunciation error, higher naturalness and speaker-similarity preference, and differing emotion-transfer difficulty across speaker conditions.
3.2. Objective Evaluation
- Fixing phone, tone or stress, language embeddings, and the encoder achieves the least pronunciation errors under extremely imbalanced 1-minute adaptation data.The evaluation used 30 abnormal or long sentences totaling 4000 words, scored by 50–80 untrained subjects across sessions.Source: §3.2 · p. 3
3.3.1. Speaker Adaptation
- The experiments compare SMA and AdaDurIAN with 1, 3, and 20 minutes of adaptation data, using MOS and ABX speaker-similarity tests on held-out Chinese and English sentences.Each test speaker has separate datasets at those durations, and the tests assess naturalness and resemblance to a recorded reference.Source: §3.3.1 · pp. 3–4, §3 · p. 3
- AdaDurIAN has higher MOS than SMA for Chinese and English sentences, including MOS 4.21 for a CN speaker with 3 minutes of data.With 1 minute, the CN speaker reaches MOS 4.14 on Chinese and 4.13 on English; the authors report cross-lingual generalization with few minutes of monolingual data.Source: §3.3.1 · p. 4
- AdaDurIAN gains more speaker-similarity preferences than SMA for the CN speaker and significantly outperforms SMA on English sentences for the EN speaker.For the EN speaker on Chinese sentences, AdaDurIAN remains comparable with SMA.Source: §3.3.1 · p. 4, Table 3 · p. 4
3.3.2. Emotion Transfer
- Emotion-transfer accuracy is higher for female-to-female than female-to-male transfer: 64% versus 52% mean emotion-classification accuracy.Neutral and happiness transfer are easiest, sadness is intermediate, and anger is hardest.Source: §3.3.2 · p. 4, Table 4 · p. 4, Table 5 · p. 4
4. Conclusions
The paper concludes that AdaDurIAN is a few-shot adaptive neural TTS system designed for naturalness and speaker similarity. Its evaluations show stronger naturalness, speaker-similarity preference, cross-lingual speaking, and promising emotion-transfer performance than the comparison system.
4. Conclusions
- AdaDurIAN provides higher naturalness, more speaker-similarity preferences, and especially fluent cross-lingual speaking than a Tacotron-like model in few-shot tasks.The paper also applies AdaDurIAN to emotion transfer and reports promising performance.Source: §4 · p. 4