Source-linked AI summary
ESPnet-TTS: Unified, Reproducible, and Integratable Open Source End-to-End Text-to-Speech Toolkit
Tomoki Hayashi, Ryuichi Yamamoto, Katsuki Inoue, Takenori Yoshimura, Shinji Watanabe, Tomoki Toda, Kazuya Takeda, Yu Zhang, Xu Tan
TL;DR
E2E-TTS research needs accessible, reproducible tooling that supports modern models and broader experimentation. ESPnet-TTS addresses this with unified ASR–TTS recipes, pretrained resources, and multiple E2E-TTS architectures. Its models achieve state-of-the-art performance comparable to recent toolkits, including a 4.25 MOS on LJSpeech.
Problem
E2E-TTS research requires user-friendly, reproducible tooling supporting modern models, varied recipes, baselines, and integrated evaluation.
Method
ESPnet-TTS extends ESPnet with Tacotron 2, Transformer TTS, and FastSpeech models, unified Kaldi-style ASR–TTS recipes, pretrained models, and waveform synthesis components.
Results
MOS 4.25 on the LJSpeech dataset, with ESPnet-TTS models achieving state-of-the-art performance comparable to other latest toolkits.
Takeaways & Limitations
The unified toolkit supports reproducible baselines, demonstrations, ASR-based TTS evaluation, and research combining ASR and TTS.
Takeaways & Limitations
FastSpeech experiments omitted knowledge distillation, which the authors expect to affect voice quality and plan to add in future work.
Abstract
from arXiv · showhide
This paper introduces a new end-to-end text-to-speech (E2E-TTS) toolkit named ESPnet-TTS, which is an extension of the open-source speech processing toolkit ESPnet. The toolkit supports state-of-the-art E2E-TTS models, including Tacotron~2, Transformer TTS, and FastSpeech, and also provides recipes inspired by the Kaldi automatic speech recognition (ASR) toolkit. The recipes are based on the design unified with the ESPnet ASR recipe, providing high reproducibility. The toolkit also provides pre-trained models and samples of all of the recipes so that users can use it as a baseline. Furthermore, the unified design enables the integration of ASR functions with TTS, e.g., ASR-based objective evaluation and semi-supervised learning with both ASR and TTS models. This paper describes the design of the toolkit and experimental evaluation in comparison with other toolkits. The experimental results show that our models can achieve state-of-the-art performance comparable to the other latest toolkits, resulting in a mean opinion score (MOS) of 4.25 on the LJSpeech dataset. The toolkit is publicly available at https://github.com/espnet/espnet.
1. INTRODUCTION
ESPnet-TTS extends ESPnet with reproducible, user-friendly support for state-of-the-art E2E-TTS models, multilingual recipes, pretrained baselines, and ASR–TTS integration. It is designed to accelerate E2E-TTS research while simplifying training and evaluation.
- E2E-TTS systems can be trained directly from text–speech pairs without complex language-specific front ends or hand-annotated phoneme alignments.
- ESPnet-TTS supports Tacotron 2, Transformer TTS, and FastSpeech alongside Kaldi-style recipes unified with ESPnet ASR procedures.
- Recipes cover more than ten languages, single- and multi-speaker TTS, and speaker adaptation, with procedures designed to reproduce results.
- Pretrained models and generated samples for all recipes let users establish baselines or demonstrate TTS systems directly.
- Unified ASR–TTS design enables ASR-based CER evaluation that can automatically identify alignment failures such as repetitions and deletions.
2. RELATED WORK
ESPnet-TTS is positioned against conventional SPSS and open-source E2E-TTS toolkits by combining broader model, recipe, pretrained-resource, and ASR-integration support. The comparison emphasizes reproducibility and coverage beyond the limited offerings of other toolkits.
- HTS targets HMM/DNN-based SPSS, while Merlin supports DNN-based SPSS with multiple neural-network architectures.
- The comparison includes six well-maintained GitHub E2E-TTS toolkits selected for having more than 1000 stars.
- ESPnet-TTS provides Tacotron 2, Transformer TTS, and FastSpeech, plus reproducible recipes for more than ten languages, multi-speaker TTS, and speaker adaptation.
- Pretrained models and samples are supplied for all recipes, supporting baseline research and TTS demonstrations.
- Unified ASR and TTS recipes support ASR-based objective evaluation and semi-supervised learning that combines both modules.
3. FEATURES OF ESPNET-TTS
ESPnet-TTS combines E2E-TTS models with reproducible Kaldi-style recipes, shared ASR/TTS processing, synthesis options, and ASR-integrated functions. Its features span model support, training and waveform generation, recipe interoperability, objective evaluation, and advanced ASR–TTS learning.
- Core toolkit: The toolkit contains a PyTorch library of E2E-TTS models and all-in-one Bash recipes covering complete experiments.The recipes follow the Kaldi style.
- Models: ESPnet-TTS supports Tacotron 2, Transformer TTS, and FastSpeech, mapping character or phoneme sequences to acoustic-feature sequences.The output includes features such as log Mel-filter bank features.
- Models: Tacotron 2 uses recurrent sequence-to-sequence modeling, Transformer TTS uses multi-head self-attention, and FastSpeech enables non-autoregressive generation through teacher-student training.The toolkit also supports forward attention variants for Tacotron 2.
- Synthesis: Generated acoustic features can be converted into speech with Griffin–Lim, WaveNet vocoder, or Parallel WaveGAN.WaveNet improves naturalness but generates slowly, whereas Parallel WaveGAN generates faster than real time with comparable quality.
- Recipe design and ASR integration: The recipes share ASR and TTS data formats across stages −1 to 2, enabling interconversion between the two recipe types.The unified design also supports ASR-based CER evaluation for detecting deletions and repetitions in generated speech.
- Recipe design and ASR integration: ASR–TTS integration includes ASR-based objective evaluation, cycle-consistency training, and semi-supervised training using both ASR and TTS modules.These advanced recipes are intended to support further end-to-end processing research.
4. EXPERIMENTAL EVALUATION
The evaluation compares ESPnet-TTS models using ASR-based CER, feature-generation speed, and subjective naturalness on LJSpeech. FastSpeech provides the strongest objective speed and error results, while phoneme-based Transformer TTS achieves the best reported naturalness among evaluated models.
- Experimental setup: The evaluation used LJSpeech, with 12,600 utterances for training, 250 for validation, and 250 for evaluation.The dataset contains 24 hours of English speech from a single speaker.
- Objective evaluation: FastSpeech.v2 achieved the best CER result, reducing substitution, deletion, and insertion errors relative to its Transformer.v1 teacher.FastSpeech.v3 performed worse than FastSpeech.v2, especially on deletion errors.
- Objective evaluation: All models generated features below RTF = 1.0 on CPU, while FastSpeech was 30 times faster than Tacotron 2 and 200 times faster than Transformer TTS on GPU.The evaluation used 16 CPU threads and one NVIDIA TITAN V GPU; the speed advantage is attributed to FastSpeech’s non-autoregressive architecture.
- Subjective evaluation: The subjective evaluation compared ESPnet-TTS models with Merlin, NVIDIA, and Mozilla toolkit models using MOS for naturalness.The comparison included a conventional SPSS system and pre-trained Tacotron 2 systems with neural vocoders.
- Subjective evaluation: Phoneme-based Transformer TTS v3 outperformed all evaluated models in naturalness, while Tacotron 2 and Transformer TTS had nearly identical naturalness performance.The subjective evaluation used audio samples presented to listeners instructed to use headphones in a quiet room.
5. SUMMARY
ESPnet-TTS is an open-source extension of ESPnet that unifies reproducible E2E-TTS recipes with ASR-compatible design. Its models achieved performance comparable to recent toolkits, including a MOS of 4.25 on LJSpeech.
- Summary: ESPnet-TTS supports state-of-the-art E2E-TTS models and recipes unified with ASR recipes for high reproducibility.The toolkit was developed to make E2E-TTS research more user-friendly and accelerate the field.
- Summary: The experimental evaluation found that ESPnet-TTS models achieved state-of-the-art performance comparable to other recent toolkits.The reported result includes MOS evaluation on the LJSpeech dataset.
- Summary: MOS reached 4.25 on the LJSpeech dataset.