Source-linked AI summary
ESPnet-ST: All-in-One Speech Translation Toolkit
Hirofumi Inaguma, Shun Kiyono, Kevin Duh, Shigeki Karita, Nelson Enrique Yalta Soplin, Tomoki Hayashi, Shinji Watanabe
TL;DR
Speech-translation research lacks a straightforward way to develop end-to-end and cascaded systems across incompatible pipelines. ESPnet-ST unifies ASR, MT, TTS, and ST code, models, and reproducible recipes, and reports improved vanilla Transformer performance with multi-task learning and transfer learning.
Problem
Existing toolkits supporting ASR and MT are not always straightforward to use for E2E-ST and Cascade-ST because their training and inference pipelines are incompatible.
Method
ESPnet-ST provides a unified ASR/MT/ST codebase with benchmark recipes covering preprocessing, feature extraction, training, and inference, alongside multilingual training support.
Results
Both multi-task learning and transfer learning improved the performance of the vanilla Transformer.
Takeaways & Limitations
The toolkit is intended to make research and reproduction across end-to-end and cascaded speech translation approaches easier.
Takeaways & Limitations
Existing frameworks can have incompatible training and inference pipelines, limiting straightforward use for E2E-ST and Cascade-ST.
Abstract
from arXiv · showhide
We present ESPnet-ST, which is designed for the quick development of speech-to-speech translation systems in a single framework. ESPnet-ST is a new project inside end-to-end speech processing toolkit, ESPnet, which integrates or newly implements automatic speech recognition, machine translation, and text-to-speech functions for speech translation. We provide all-in-one recipes including data pre-processing, feature extraction, training, and decoding pipelines for a wide range of benchmark datasets. Our reproducible results can match or even outperform the current state-of-the-art performances; these pre-trained models are downloadable. The toolkit is publicly available at https://github.com/espnet/espnet.
1 Introduction
Speech translation research spans cascaded and end-to-end systems, but existing toolkits make these approaches difficult to develop together. ESPnet-ST addresses this gap with a unified, all-in-one framework containing shared code, recipes, and models across speech-processing tasks.
- Motivation: Speech translation converts speech in one language into text in another, using either cascaded ASR–MT modules or a direct end-to-end mapping.Cascaded systems may include text normalization, while E2E-ST uses a single sequence-to-sequence model.
- Motivation: E2E-ST can reduce inference latency, support end-to-end back-propagation, and serve settings where source transcripts are unavailable or difficult to obtain.The paper notes that translation-quality comparisons between E2E-ST and Cascade-ST remain unsettled.
- Motivation: Existing ASR and MT toolkits are not always straightforward to use for E2E-ST and Cascade-ST because their training and inference pipelines are incompatible.This motivates a unified framework for developing and comparing both approaches.
- ESPnet-ST: ESPnet-ST extends ESPnet with a unified codebase for ASR, MT, TTS, and ST, plus reproducible recipes covering a wide range of speech-translation benchmarks.Recipes encapsulate complete training and inference procedures, while the shared framework is intended to simplify customization.
- ESPnet-ST: The toolkit provides open-source code, downloadable pretrained models, and an interactive speech-to-speech translation demo hosted by Google Colab.Its recipes cover corpora including Fisher-CallHome, Libri-trans, How2, and Must-C.
2 Design
ESPnet-ST organizes reproducible experiments around corpus-specific task directories, automated tool setup, and single-script execution. Configuration files and experiment outputs are separated into dedicated directories.
- Tool setup: Required neural-network, ASR, and MT tools are automatically downloaded and built under tools by a make command.The implementation uses a PyTorch backend and includes PyTorch, Kaldi, Moses, and sentencepiece.
- Recipes: Corpus directories in egs contain task directories such as st1, where run.sh executes experiments.This structure supports reproducing baseline systems with a single script.
- Configuration: Configuration YAML files in conf specify feature extraction, data augmentation, model training, and decoding settings.
- Outputs: Model directories and checkpoints are saved under exp.
2.3 Tasks
ESPnet-ST supports ASR, MT, ST, LM, and TTS within a unified toolkit and executes experiments through staged preprocessing, feature extraction, training, decoding, and evaluation pipelines. Its recipes include both end-to-end and cascade speech translation workflows.
- Supported tasks: ESPnet-ST supports LM, TTS, ASR, ST, and MT tasks in a single toolkit.The toolkit also provides examples for combining ASR and MT modules for Cascade-ST.
- Pipeline: The processing pipeline follows corpus-dependent preprocessing, feature extraction, training, decoding, and evaluation stages.It uses Kaldi-style data preparation to facilitate speech-data augmentation from resources in egs.
- Preprocessing: Preprocessing normalizes and tokenizes transcriptions and translations, producing truecased, lowercased, and punctuation-removed variants.The punctuation-removed lowercased variant is designed for conventional ASR systems.
- Stages: The recipes extract speech features, dump dataset JSON files, optionally train an LM for ASR, train RNN or Transformer models, and perform model averaging, beam search, and scoring.The JSON files contain token, utterance, speaker, and language IDs, sequence lengths, transcriptions, and translations.
- Cascade-ST: In the Cascade-ST recipe, evaluation feeds ASR outputs into the MT model.
2.5 Multi-task learning and transfer learning
ESPnet-ST addresses the difficulty of optimizing end-to-end speech translation by offering auxiliary ASR and MT objectives and initialization from pretrained ASR and MT components. The authors also report an ASR-performance trade-off for one truecasing option.
- Motivation: End-to-end speech translation is harder to optimize than separately trained ASR and MT models.The toolkit therefore exposes multitask-learning and transfer-learning options.
- Multi-task learning: E2E-ST training can add auxiliary ASR and MT objectives during Stage 4.
- Caveat: Truecased ASR training was found to degrade ASR performance.The paper reports this as a footnote associated with the text-treatment options.
- Transfer learning: The ST encoder can be initialized from a pretrained ASR encoder, while the ST decoder can be initialized from a pretrained MT decoder.These initializations are available through the asr1 and mt1 recipes.
2.6 Speech data augmentation
ESPnet-ST incorporates speech-data augmentation techniques intended to improve robustness and stabilize end-to-end speech-translation training. These include speed perturbation and SpecAugment.
- Robustness: The toolkit implements robustness techniques developed for the ASR component.
- Speed perturbation: Speed perturbation changes speech speed by factors of 0.9, 1.0, and 1.1, producing 3-fold data augmentation.The authors found speed perturbation important for stabilizing E2E-ST training.
- SpecAugment: SpecAugment randomly masks time and frequency blocks in log mel-filterbank features.The paper states that it improves ASR performance and is effective for E2E-ST.
2.7 Multilingual training
Multilingual training combines datasets from different language pairs to train a single speech-to-text translation model, and ESPnet-ST supports this approach in several recipes.
- Multilingual training combines datasets from different language pairs to train a single model.
- ESPnet-ST supports multilingual E2E-ST and MT models in several recipes.
2.8 Additional features
ESPnet-ST adds infrastructure for managing experiments, scaling training and decoding, monitoring performance, and combining models during decoding.
- Experiment manager: The experiment manager customizes data loading, training, and evaluation while sharing common processes across tasks.
- Large-scale training/decoding: Training and decoding support job schedulers, multiple GPUs, and half- or mixed-precision computation.
- Performance monitoring: TensorBoard collectively monitors attention weights, training and validation scores, and losses for ASR, MT, and ST.
- Ensemble decoding: Beam-search decoding supports averaging posterior probabilities from multiple models.
3 Example Models
The example ESPnet-ST models combine Transformer-based ASR, MT, and E2E-ST components, with transfer learning and optional multi-task objectives, and can connect translation to TTS.
- Automatic speech recognition (ASR): The example ASR component uses a Transformer-based hybrid CTC/attention framework with an external LSTM language model during decoding.
- Machine translation (MT): The MT component uses a six-block Transformer encoder-decoder trained on lowercased, punctuation-removed source sentences.
- End-to-end speech translation (E2E-ST): The E2E-ST model combines a speech encoder and translation decoder, initializes them from pretrained ASR and MT parameters, and optionally adds multi-task objectives.
- Text-to-speech (TTS): ESPnet-ST supports TTS after speech translation, using feature-generation and vocoder networks with architectures including Tacotron 2, Transformer-TTS, and FastSpeech.
- The example system variants include Transformer models with ASR/MT transfer, pretraining and adaptors, SpecAugment, encoder or decoder initialization, and three-model ensembles.
4 Experiments
ESPnet-ST recipes are evaluated across multiple speech-translation benchmarks and an MT benchmark, with reported results matching or exceeding prior systems while reducing training or inference cost in some settings.
- Experiments cover Fisher-CallHome Spanish, Libri-trans English-French, How2 English-Portuguese, Must-C across eight directions, and IWSLT16 English-German MT.
- The experiments use Moses tokenization, joint BPE vocabularies, 4-gram BLEU, and 83-dimensional log-mel-plus-pitch speech features.
- Fisher-CallHome Spanish: On Fisher-CallHome Spanish, multi-task learning and transfer learning improve vanilla Transformer, while the best SpecAugment system matches current state-of-the-art performance without additional data.
- Libri-trans: On Libri-trans, all models use the same data resource and are competitive with previous work.
- How2: On How2, the systems significantly outperform the previous RNN-based model, and the interactive demo reports an inference real-time factor of 0.7755.
- Must-C: On Must-C, the systems outperform previous work implemented with customized Fairseq by a large margin.
- IWSLT16 MT: On IWSLT16 MT, ESPnet-ST performs almost comparably to Fairseq, with the gap attributed to minor implementation differences and hyperparameter sensitivity.
5 Conclusion
ESPnet-ST supports fast development of both end-to-end and cascaded speech-translation systems. Its all-in-one scripts cover corpus-specific preprocessing, feature extraction, training, and inference.
- ESPnet-ST targets fast development of end-to-end and cascaded speech-translation systems.
- Its example scripts integrate corpus-dependent preprocessing, feature extraction, training, and inference.
- The toolkit plans to expand corpus support and implement techniques bridging end-to-end and cascaded approaches.