Source-linked AI summary
MultiSpeech: Multi-Speaker Text to Speech with Transformer
Mingjian Chen, Xu Tan, Yi Ren, Jin Xu, Hao Sun, Sheng Zhao, Tao Qin, Tie-Yan Liu
TL;DR
Transformer’s parallel computation makes text-speech alignment difficult, especially with noisy, diverse multi-speaker data. MultiSpeech combines attention, encoder, and decoder techniques to improve alignment, and improves voice quality over naive Transformer TTS on VCTK and LibriTTS.
Problem
Transformer’s parallel computation makes text-speech alignment difficult, and noisy, diverse multi-speaker speech further complicates alignment.
Source: §1 · p. 1Method
MultiSpeech combines a diagonal attention constraint, encoder layer normalization before adding position embeddings, and a decoder pre-net bottleneck.
Source: §1 · p. 1Results
MultiSpeech gains 1.01 MOS on VCTK and 1.46 MOS on LibriTTS over naive Transformer TTS, and a MultiSpeech teacher yields multi-speaker FastSpeech without quality degradation.
Source: §1 · p. 2Takeaways & Limitations
On VCTK and LibriTTS, MultiSpeech provides higher-quality multi-speaker voice than naive Transformer TTS, while its teacher model supports extremely fast-inference FastSpeech without quality degradation.
Source: §1 · p. 2The encoder normalization is not applied to the decoder because mel-spectrogram input is not learnable and is usually normalized to a fixed range.
Source: §4.1 · p. 3
Abstract
from arXiv · showhide
Transformer-based text to speech (TTS) model (e.g., Transformer TTS~\cite{li2019neural}, FastSpeech~\cite{ren2019fastspeech}) has shown the advantages of training and inference efficiency over RNN-based model (e.g., Tacotron~\cite{shen2018natural}) due to its parallel computation in training and/or inference. However, the parallel computation increases the difficulty while learning the alignment between text and speech in Transformer, which is further magnified in the multi-speaker scenario with noisy data and diverse speakers, and hinders the applicability of Transformer for multi-speaker TTS. In this paper, we develop a robust and high-quality multi-speaker Transformer TTS system called MultiSpeech, with several specially designed components/techniques to improve text-to-speech alignment: 1) a diagonal constraint on the weight matrix of encoder-decoder attention in both training and inference; 2) layer normalization on phoneme embedding in encoder to better preserve position information; 3) a bottleneck in decoder pre-net to prevent copy between consecutive speech frames. Experiments on VCTK and LibriTTS multi-speaker datasets demonstrate the effectiveness of MultiSpeech: 1) it synthesizes more robust and better quality multi-speaker voice than naive Transformer based TTS; 2) with a MutiSpeech model as the teacher, we obtain a strong multi-speaker FastSpeech model with almost zero quality degradation while enjoying extremely fast inference speed.
1. Introduction
Transformer TTS offers parallel computation, but learning text-to-speech alignment is difficult in multi-speaker settings with noisy, diverse speech. MultiSpeech introduces alignment-focused techniques and improves voice quality over naive Transformer TTS on VCTK and LibriTTS.
1. Introduction
- Multi-speaker TTS commonly relies on noisy, low-quality ASR or voice-conversion recordings because collecting clean studio data for hundreds or thousands of speakers is costly.Such recordings vary in prosody, accent, speaking speed, and recording conditions.Source: §1 · p. 1
- Transformer’s parallel training prevents using prior decoder-step attention strategies, while noisy multi-speaker speech makes alignment harder than in single-speaker TTS.Existing multi-speaker systems mostly use RNN or CNN backbones, with few Transformer-based attempts.Source: §1 · p. 1
- MultiSpeech constrains encoder-decoder attention toward diagonal alignments, normalizes phoneme embeddings before adding positional embeddings, and bottlenecks the decoder pre-net to discourage copying adjacent frames.The caption identifies green blocks in Figure 1 as modules newly added to Transformer-based multi-speaker TTS.Source: §1 · pp. 1–2, Figure 1 · p. 2
- MultiSpeech gains 1.01 MOS on VCTK and 1.46 MOS on LibriTTS over naive Transformer TTS, while synthesizing robust, high-quality multi-speaker voice.A MultiSpeech teacher also yields a strong multi-speaker FastSpeech model without quality degradation and with extremely fast inference.Source: §1 · p. 2
2. Background
Transformer TTS uses self-attention and feed-forward blocks, with a decoder pre-net and output layers for mel-spectrogram and stop prediction. Its parallel computation can make text-speech alignment difficult, motivating alignment-focused modifications for multi-speaker TTS.
2. Background
- Transformer TTS uses multi-head self-attention and feed-forward networks in each block, plus a decoder pre-net and mel and stop prediction layers.Figure 1 depicts the MultiSpeech structure, with green blocks marking its newly added modules.Source: §2 · p. 2, Figure 2 · p. 2
- Parallel computation during Transformer training harms text-speech attention alignment, complicating multi-speaker TTS under diverse acoustic conditions.The paper analyzes Transformer components and proposes corresponding alignment modifications.Source: §2 · p. 2
- Prior multi-speaker systems use RNN, CNN, or VAE-based methods, while this work builds on Transformer for its parallel training and sequence modeling advantages.Related alignment work includes location-sensitive attention and speaker-conditioned position encoding [7].Source: §2 · p. 2
3. Improving Text-to-Speech Alignment
MultiSpeech addresses alignment from attention, encoder, and decoder components. Its techniques constrain attention toward diagonal paths, preserve positional information in encoder inputs, and reduce decoder-frame copying.
3.1. Diagonal Constraint in Attention
- The diagonal constraint rewards attention weight inside a diagonal region, with its shape set by speech and text lengths and a bandwidth hyperparameter.The loss is the negative diagonal attention rate and is added to the TTS loss with weight λ; at inference, a sliding window restricts attention and moves according to the attention centroid.Source: §3.1 · p. 3
- The inference sliding-window strategy attends to one previous and four future phonemes, moving its center after the attention centroid deviates for three consecutive frames.Compared with [7], it includes the previous position and uses the attention centroid rather than the highest-weight position, helping avoid premature movement that can skip phonemes and speed speech.Source: §3.1 · p. 3
3.2. Position Information in Encoder
- Because phoneme-embedding scale can overwhelm or underweight fixed-scale positional embeddings, the encoder applies layer normalization to phoneme embeddings before adding position embeddings.The input becomes LN(x) + p, restricting phoneme-embedding scale through learned scale and bias parameters.Source: §3.2 · p. 3
3.3. Pre-Net Bottleneck in Decoder
- Adjacent mel-spectrogram frames overlap substantially, so the autoregressive decoder may copy the current frame instead of learning from text.The proposed decoder pre-net reduces the bottleneck to 1/8 of the original hidden size, which the authors found essential for meaningful alignment and avoiding direct copying.Source: §3.3 · p. 3
4. Experiments and Results
Experiments evaluate MultiSpeech on VCTK and LibriTTS, then examine its alignment techniques and use as a FastSpeech teacher. The reported comparisons cover voice quality, alignment measures, and the distilled model’s MOS.
4.1. Experimental Setup
- Experiments use VCTK (44 hours, 108 speakers) and LibriTTS (586 hours, 2456 speakers), evaluating voice quality with MOS and alignment with diagonal attention rate r.Both corpora are converted to 16 kHz audio; evaluation uses six speakers per dataset.Source: §4.1 · pp. 3–4
4.2. The Quality of MultiSpeech
- MultiSpeech achieves higher MOS than naive multi-speaker Transformer TTS, whose alignments often fail and produce skipped or repeated words or crashed voice.On VCTK, its MOS is also close to GT mel + Vocoder.Source: §4.2 · p. 4, Table 1 · p. 4
4.3. Method Analysis
- Removing DC, LN, or PB lowers both MOS and diagonal rate r; removing all three causes larger drops, supporting their contribution to alignment and voice quality.The ablations are conducted on VCTK.Source: §4.3 · p. 4, Table 2 · p. 4
- LN yields the highest diagonal rate r among LN, learnable-weight, and baseline settings, while the learnable-weight setting yields the lowest.The reported similarity for LN falls between learnable weight and baseline.Source: §4.3 · p. 4, Table 3 · p. 4
4.4. Extension on FastSpeech
- A MultiSpeech teacher produces a multi-speaker FastSpeech model with nearly the same MOS on VCTK.MOS evaluation uses six speakers, three men and three women, with ten sentences each.Source: §4.4 · p. 4, Table 4 · p. 4
5. Conclusions
The conclusion presents MultiSpeech as a multi-speaker Transformer TTS system using three alignment techniques. The experiments support improved voice quality and stability over the Transformer baseline, while teacher-based FastSpeech offers extremely fast inference.
5. Conclusions
- MultiSpeech combines diagonal attention constraint, encoder layer normalization, and decoder pre-net bottleneck to improve multi-speaker text-to-speech alignment.The conclusion reports higher-quality and more stable voice than the Transformer TTS baseline.Source: §5 · p. 4
- The learnable positional weight reaches α = 2.62, compared with about 0.5 in the single-speaker setting.The authors conjecture that it learns a large global weight to highlight positional embeddings.Source: §5 · p. 4
- The authors suggest that using more unlabeled text in knowledge distillation could raise FastSpeech quality to match or outperform its teacher.Source: §5 · p. 4