Source-linked AI summary
DeviceTTS: A Small-Footprint, Fast, Stable Network for On-Device Text-to-Speech
Zhiying Huang, Hao Li, Ming Lei
TL;DR
On-device TTS needs high-quality speech without the large, complex models that hinder deployment. DeviceTTS addresses this with duration-based alignment, DFSMN components, and a mix-resolution decoder, achieving comparable performance with Tacotron and FastSpeech using 1.4 M parameters and 0.099 GFLOPS. The authors conclude that it can meet the needs of most devices in practical applications.
Problem
Large and complex neural TTS models are difficult to deploy on devices despite improving synthesized speech quality.
Source: §1 · p. 1Method
DeviceTTS uses a duration predictor, DFSMN components, and a mix-resolution decoder with autoregressive multi-frame prediction followed by refinement.
Source: §1 · p. 1, §2 · p. 2, §2.1 · p. 2Results
1.4 M model parameters and 0.099 GFLOPS yield performance comparable with Tacotron and FastSpeech using WORLD and LPCNet vocoders.
Source: §1 · p. 1, §4 · p. 4, §3.3.1 · p. 4Takeaways & Limitations
DeviceTTS can meet the needs of most devices in practical on-device TTS applications.
Source: §3.3.2 · p. 4, §4 · p. 4
Abstract
from arXiv · showhide
With the number of smart devices increasing, the demand for on-device text-to-speech (TTS) increases rapidly. In recent years, many prominent End-to-End TTS methods have been proposed, and have greatly improved the quality of synthesized speech. However, to ensure the qualified speech, most TTS systems depend on large and complex neural network models, and it's hard to deploy these TTS systems on-device. In this paper, a small-footprint, fast, stable network for on-device TTS is proposed, named as DeviceTTS. DeviceTTS makes use of a duration predictor as a bridge between encoder and decoder so as to avoid the problem of words skipping and repeating in Tacotron. As we all know, model size is a key factor for on-device TTS. For DeviceTTS, Deep Feedforward Sequential Memory Network (DFSMN) is used as the basic component. Moreover, to speed up inference, mix-resolution decoder is proposed for balance the inference speed and speech quality. Experiences are done with WORLD and LPCNet vocoder. Finally, with only 1.4 million model parameters and 0.099 GFLOPS, DeviceTTS achieves comparable performance with Tacotron and FastSpeech. As far as we know, the DeviceTTS can meet the needs of most of the devices in practical application.
1. INTRODUCTION
DeviceTTS targets on-device TTS by combining a compact DFSMN-based architecture with duration-based alignment and a mix-resolution decoder. It is motivated by deployment constraints and instability or generalization issues in existing systems.
Motivation
- Large and complex neural TTS models improve speech quality but are difficult to train and do not support real-time synthesis.This creates a deployment challenge for on-device applications.Source: §1 · p. 1
Existing systems
- Tacotron simplifies the TTS pipeline and improves naturalness, but poorly learned attention alignments can cause words to be skipped or repeated.Model compression experiments also found that reducing Tacotron size harms robustness, possibly because alignment requires sufficient parameters.Source: §1 · p. 1
- FastSpeech uses a duration predictor to avoid Tacotron’s skipping and repeating problem, but showed weaker generalization for long utterances beyond the training-set maximum length.The authors suspect that global self-attention contributes to confusion on unseen length patterns.Source: §1 · p. 1
DeviceTTS
- DeviceTTS uses DFSMN components, a duration predictor, and a mix-resolution decoder to provide a small-footprint, fast, stable on-device TTS network.The duration predictor bridges encoder and decoder, while the mix-resolution decoder balances inference speed with speech quality.Source: §1 · p. 1
- The paper combines a DFSMN-based design with a mix-resolution decoder that predicts multiple acoustic frames before refining them with single-frame prediction.The paper evaluates DeviceTTS with WORLD and LPCNet vocoders.Source: §1 · p. 1
2. DEVICETTS
DeviceTTS combines a duration-regulated encoder–decoder with compact DFSMN blocks and a mix-resolution decoder for efficient, stable on-device speech synthesis.
2. DEVICETTS
- DeviceTTS uses an encoder, duration predictor, length regulator, and decoder to expand text representations into acoustic features.The duration predictor supplies frame counts, and the length regulator expands encoder outputs before decoding.Source: §2 · p. 2
- DeviceTTS training uses a combined acoustic-feature and phone-duration loss, with MAE applied to both components.The combined objective is L = Laco + Ldur.Source: §2 · p. 2
2.1. DFSMN block
- DFSMN blocks provide compact sequential modeling across the encoder, duration predictor, and decoder without recurrent feedback.Memory blocks encode historical context, while skip connections help mitigate gradient vanishing in deep networks.Source: §2.1 · p. 2
- DFSMN latency depends on look-ahead order, so its latency-control window supports bounded context and more stable real-time synthesis.The look-back and look-ahead filters determine the modeled context.Source: §2.1 · p. 2
2.2. mix-resolution decoder
- The mix-resolution decoder combines autoregressive multi-frame prediction with a refine network for finer-grained acoustic modeling.Selected length-regulated frames are concatenated with Prenet outputs, then reshaped for single-frame refinement.Source: §2.2 · p. 3
3.1. Speech corpus
The experiments use a prosodically and phonetically rich Mandarin corpus with approximately 30 hours of speech and phone-level timing annotations.
3.1. Speech corpus
- The corpus contains approximately 30,000 utterances from a female native Mandarin speaker, totaling about 30 hours at 24kHz.It includes prosody annotations and phone-level timestamps.Source: §3.1 · p. 3
3.2. Experiment setup
The evaluation compares DeviceTTS with Tacotron and FastSpeech using two lightweight vocoders, controlled training settings, and subjective naturalness testing.
3.2.1. Model Configuration
- The study uses FastSpeech and Tacotron as baselines alongside DeviceTTS, with FastSpeech configured using six FFT blocks on both sides.Tacotron uses a structure similar to a published baseline.Source: §3.2.1 · p. 3
3.2.2. Vocoder
- DeviceTTS is evaluated with WORLD and LPCNet vocoders selected for high-quality, stable, lightweight waveform reconstruction.WORLD uses 67-dimensional acoustic features, while LPCNet uses 23-dimensional features.Source: §3.2.2 · p. 3
- The multi-frame output parameter is set to 8 with WORLD and 3 with LPCNet to account for their different window shifts.DeviceTTS and Tacotron both use multi-frame prediction.Source: §3.2.2 · p. 3
3.2.3. Training and Test
- Training runs for 300k steps to convergence with Adam on one NVIDIA V100 GPU, using batch sizes of 32, 16, and 32 for Tacotron, FastSpeech, and DeviceTTS.The optimizer uses β1 = 0.9, β2 = 0.999, lrate = 0.002, and ϵ = 10^-9.Source: §3.2.3 · p. 3
- Naturalness is measured on 100 test utterances with nine-scale MOS ratings from about 200 listeners, while FastSpeech tests are capped at its training-set maximum length.Each utterance is rated five times by different listeners.Source: §3.2.3 · p. 3
3.3. Results
DeviceTTS delivers naturalness comparable to Tacotron and FastSpeech while substantially reducing model size and computational complexity for on-device TTS.
3.3.1. Audio Quality
- DeviceTTS achieves comparable naturalness to Tacotron and FastSpeech, with MOS values of 4.193 using WORLD and 4.255 using LPCNet.Tacotron has the best naturalness MOS, while FastSpeech is slightly lower than Tacotron by an average MOS gap of -0.004.Source: §3.3.1 · p. 4
3.3.2. Complexity
- 1.4 M parameters and 0.099 GFLOPS give DeviceTTS lower model and computational complexity than Tacotron and FastSpeech.Tacotron uses 13.5 M parameters and 0.491 GFLOPS, while FastSpeech uses 36.1 M parameters and 8.58 GFLOPS.Source: §3.3.2 · p. 4
3.4. Ablation Study
Ablation studies show that the MR decoder improves naturalness, while the AR network preserves similar quality with lower computational cost than its non-AR alternative.
3.4.1. MR decoder
- Removing the MR decoder reduces CMOS by -0.254, supporting the effectiveness of the finer-grained acoustic-feature prediction structure.The comparison uses a similar number of parameters for DeviceTTS with and without the MR decoder.Source: §3.4.1 · p. 4
3.4.2. AR network
- The AR and non-AR networks have nearly equal naturalness, with the non-AR version differing by -0.003 CMOS.The non-AR alternative replaces the AR network with DFSMN blocks.Source: §3.4.2 · p. 4
- The AR network uses about 60% less GFLOPS than the non-AR network for generating the first output frame.The non-AR network requires a larger look-ahead order to maintain speech quality, increasing latency.Source: §3.4.2 · p. 4
4. CONCLUSION
DeviceTTS is presented as a lightweight, fast, stable on-device TTS network that combines a duration predictor, DFSMN blocks, and an MR decoder. It achieves comparable performance with Tacotron and FastSpeech using 1.4 M parameters and 0.099 GFLOPS, while future work targets faster vocoders and multi-speaker operation.
4. CONCLUSION
- DeviceTTS combines a duration predictor, DFSMN blocks, and an MR decoder to provide lightweight, fast, stable on-device TTS.The duration predictor bridges the encoder and decoder to improve stability.Source: §4 · p. 4
- 1.4 M model parameters and 0.099 GFLOPS yield performance comparable with Tacotron and FastSpeech.The paper states that this configuration can meet the needs of most devices in practical application.Source: §4 · p. 4
- Future work will investigate faster vocoders such as MelGAN and multi-speaker DeviceTTS.Source: §4 · p. 4