Source-linked AI summary
Efficiently Trainable Text-to-Speech System Based on Deep Convolutional Networks with Guided Attention
Hideyuki Tachibana, Katsuya Uenoyama, Shunsuke Aihara
TL;DR
The paper addresses the training cost of recurrent neural TTS and presents Deep Convolutional TTS, a fully convolutional alternative with guided attention. DCTTS trained for approximately 15 hours on an ordinary two-GPU gaming PC, achieving almost acceptable speech quality, though audio quality remained below reported Tacotron levels.
Problem
Training recurrent components in neural TTS can require very powerful hardware or several days or weeks, motivating a faster alternative that remains accessible to small teams.
Method
DCTTS uses a fully convolutional sequence-to-sequence TTS architecture and a rapidly trained guided-attention module.
Results
2.71±0.66 MOS for DCTTS after 15 hours versus 2.07±0.62 for open Tacotron, under a non-strict comparison with different frameworks and machines.
Takeaways & Limitations
DCTTS can be trained overnight on an ordinary gaming PC with two GPUs while producing speech whose quality is almost acceptable.
Takeaways & Limitations
The authors report that audio quality is far from perfect, and MOS was below the level reported in Tacotron’s reference.
Abstract
from arXiv · showhide
This paper describes a novel text-to-speech (TTS) technique based on deep convolutional neural networks (CNN), without use of any recurrent units. Recurrent neural networks (RNN) have become a standard technique to model sequential data recently, and this technique has been used in some cutting-edge neural TTS techniques. However, training RNN components often requires a very powerful computer, or a very long time, typically several days or weeks. Recent other studies, on the other hand, have shown that CNN-based sequence synthesis can be much faster than RNN-based techniques, because of high parallelizability. The objective of this paper is to show that an alternative neural TTS based only on CNN alleviate these economic costs of training. In our experiment, the proposed Deep Convolutional TTS was sufficiently trained overnight (15 hours), using an ordinary gaming PC equipped with two GPUs, while the quality of the synthesized speech was almost acceptable.
1. INTRODUCTION
The paper motivates a manageable end-to-end TTS system by replacing costly recurrent components with a fully convolutional architecture and guided attention. DCTTS targets faster training while retaining acceptable speech quality.
- TTS systems should be manageable, maintainable, and extensible for speech non-specialists, individuals, and small teams.
- Traditional TTS is difficult to maintain because it integrates many domain-specific modules, including text analysis, F0 generation, spectrum generation, pause estimation, and vocoding.
- Tacotron connects text directly to spectrograms without hand-engineered parametric models, but its many recurrent units are costly to train.
- DCTTS is a fully CNN-based TTS system intended to train much faster than RNN-based neural TTS while maintaining acceptable sound quality.
- RNNs are less suited to parallel GPU computation, whereas CNN-based sequence models can train faster and sometimes outperform RNN-based techniques.
- DCTTS follows CNN-based sequence-to-sequence attention work and differs from WaveNet by synthesizing spectrograms rather than waveforms as a front-end and most of the back-end.
2. PRELIMINARY
The preliminary section defines the spectrogram representations and convolutional notation used by DCTTS. It also introduces gated activations and the network components for staged spectrogram synthesis.
- 2.1. Basic Knowledge of the Audio Spectrograms: STFT transforms audio waveforms into complex spectrograms, while inverse STFT supports waveform reconstruction; F′ and T′ denote frequency and temporal bins.
- 2.1. Basic Knowledge of the Audio Spectrograms: Mel spectrograms reduce frequency dimensionality from F′ to F, and DCTTS further reduces temporal dimensionality by selecting one frame every four frames.
- 2.2. Notation: Convolution and Highway Activation: The convolution notation records input and output channels, kernel size, and dilation, while convolution preserves sequence length through zero padding.
- 2.2. Notation: Convolution and Highway Activation: Highway activation combines a transformed ReLU path with the original input through sigmoid gating and element-wise products.
- 2.2. Notation: Convolution and Highway Activation: DCTTS uses Text2Mel to generate mel spectrograms and SSRN to convert coarse mel spectrograms into full STFT spectrograms.
3. PROPOSED NETWORK
The proposed network synthesizes speech spectrograms in two convolutional stages: Text2Mel produces a coarse mel spectrogram, and SSRN reconstructs a full-resolution spectrogram. Text2Mel encodes text and audio, combines them through attention, and trains the decoder against shifted ground truth using a composite spectrogram loss.
- Text2Mel: Text2Mel synthesizes a coarse mel spectrogram from input text using TextEnc, AudioEnc, Attention, and Audio Decoder modules.TextEnc produces key and value matrices, while AudioEnc produces a query matrix from the coarse mel spectrogram.
- Text2Mel: Attention evaluates character–mel-frame relationships and computes the seed representation R = V A for subsequent mel-spectrogram prediction.The seed R is concatenated with the encoded audio Q before decoding.
- Text2Mel: The Audio Decoder estimates a mel spectrogram from the concatenated seed R′ = [R, Q].The resultant output is trained to approximate the temporally shifted ground-truth mel spectrogram.
- Text2Mel: The spectrogram loss combines L1 loss with binary divergence and is back-propagated through the network.The loss is nonnegative and equals zero if and only if the synthesized spectrogram matches the target.
- Convolutional architecture: All network components are convolutional: dilated convolutions replace recurrent units, while causal convolutions support feedback during synthesis.TextEnc uses non-causal convolutions; AudioEnc and AudioDec use causal convolutions because decoder output is fed back to AudioEnc.
- SSRN: SSRN converts the coarse mel spectrogram into a full spectrogram by increasing frequency channels and quadrupling temporal length with two stride-2 deconvolution layers.SSRN uses non-causal convolutions because online processing is not considered.
4. GUIDED ATTENTION
The guided attention loss uses TTS’s approximate character-to-time alignment to encourage nearly diagonal attention. This improves attention learning speed and yields roughly correct alignments on both training and new texts, while heuristic correction addresses some synthesis-time failures.
- Motivation: TTS attention is constrained by the rough correspondence between character order and audio time, unlike alignment in machine translation.The expected relation is nearly linear: n ∼ at, with a ∼ N/T.
- Figure comparison: Figure 3 compares attention matrices trained with and without guided attention on an ICASSP test sentence, without using the section 4.2 heuristics.The left panel is without guided attention and the right panel is with guided attention.
- Guided attention loss: Guided attention loss penalizes attention matrices that deviate from a nearly diagonal pattern.The loss uses W_nt = 1 − exp{−(n/N − t/T)^2/2g^2} with g = 0.2 and is optimized with Lspec at equal weight.
- Effect on training: After ∼5K iterations, guided attention produced roughly correct attention on training and new input texts, compared with ∼50K iterations without it.With the loss, the term began decreasing after ∼100 iterations; without it, learning began after ∼10K iterations and remained vague at ∼50K iterations.
- Synthesis-time correction: At synthesis, attention sometimes skips letters or repeats words, so a heuristic forces the target forward when successive attention positions change outside −1 ≤ n_t − n_t−1 ≤ 3.The rule sets attention to the next character position using a Kronecker delta.
5. EXPERIMENT
The experiment trains and evaluates DCTTS on LJ Speech, comparing its efficiency and speech quality with an open Tacotron. After 15 hours, DCTTS produced almost correctly aligned attention and clear spectrograms, while achieving a higher MOS in a non-strict comparison.
- Experimental setup: ∼13K text-speech pairs totaling ∼24 hours from the LJ Speech Dataset were used without phoneme-level alignment.The data contained some reverberation and were preprocessed by expanding abbreviations and numeric expressions, decapitalizing capitals, and removing infrequent characters.
- Evaluation: DCTTS and the open Tacotron were compared using MOS ratings from 20 Harvard sentences synthesized by five methods.Crowdsourcing on Amazon Mechanical Turk collected ratings from 1 (Bad) to 5 (Excellent), with results statistically processed using crowdMOS.
- Training efficiency: ∼15 hours of training enabled 200K Text2Mel updates, with throughput of ∼3.8 minibatch/s for Text2Mel and ∼6.4 minibatch/s for SSRN.The models were trained independently and asynchronously on a household gaming PC equipped with two GPUs.
- Qualitative results: After 15 hours, the proposed method almost correctly focused attention on the correct characters and synthesized quite clear mel and full spectrograms.Figure 4 shows attention, mel, and linear STFT spectrograms for a 90-character input sentence.
- Quantitative results: 2.71±0.66 MOS for DCTTS versus 2.07±0.62 for Tacotron was observed, although the comparison was not strict because frameworks and machines differed.The reported MOS values are 95% confidence intervals; the Tacotron model was reported as trained for 12 days on a GTX 1080 Ti.
- Limitations: The MOS remained below the level reported in the original Tacotron paper, possibly because of limited iterations, reduced spectrogram input information, and reverberant training data.These are the paper’s three proposed reasons for the lower MOS.
6. SUMMARY AND FUTURE WORK
The paper presents DCTTS as a fully convolutional neural TTS system with a rapid attention-training technique. It was trained in approximately 15 hours on a two-GPU gaming PC, although its audio quality remained imperfect and future extensions were left open.
- DCTTS is a novel TTS technique based on deep convolutional neural networks, together with a technique for rapidly training its attention module.
- The proposed system was trained overnight in ∼15 hours on an ordinary gaming PC equipped with two GPUs, while synthesized speech quality was almost acceptable.
- Audio quality remained far from perfect, though the authors suggest tuning hyper-parameters and applying deep-learning techniques may improve it.
- The authors suggest that lighter neural TTS may support further speech-synthesis applications and make more integrated speech systems more feasible.