NeurST: Neural Speech Translation Toolkit

1 Introduction

• Cascade systems can propagate ASR errors, while direct end-to-end speech translation avoids intermediate ASR transcriptions and offers lower-latency deployment.The paper frames end-to-end modeling as feasible following progress in sequence-to-sequence machine translation and speech recognition.Source: §1 · p. 1
• Different datasets and inconsistent baseline results—including differing BLEU conventions—make fair comparison of speech translation approaches difficult.The paper identifies complex audio preprocessing and training practices such as pre-training and data augmentation as reasons reproducibility is challenging.Source: §1 · p. 1
• NeurST provides a toolkit for building and training end-to-end speech translation, ASR, and NMT models, with step-by-step recipes for preprocessing, training, and inference.It implements Transformer-based models and is designed specifically for speech translation, encapsulating speech-processing details for developers.Source: §1 · p. 2
• NeurST reports strong benchmarks and reproducible recipes as reliable baselines for speech translation research.The paper also claims its results surpass recent studies and ESPnet-ST, while the comparison details are not included in these passages.Source: §1 · p. 2

2.1 Design

• NeurST uses Dataset, Model, Task, and Executor components; their interfaces connect standardized inputs, task-specific transformations, model computation, and workflow execution.Users can customize Dataset and Model while sharing a Task, or reuse Executor workflows while customizing the other components.Source: §2.1 · p. 2

2.2 Features

• NeurST supports efficient computation with mixed precision, XLA, and distributed training through Horovod and Byteps.The toolkit is implemented with TensorFlow2 and PyTorch.Source: §2.2 · p. 2, §2 · p. 2
• Data preprocessing supports on-the-fly audio and text processing or preprocessed binary records, which improve training I/O performance.The toolkit also provides a command-line tool to iterate over Dataset formats, preprocess samples according to Task, and write records to disk.Source: §2.2 · pp. 2–3
• Transfer learning can initialize speech translation encoders from ASR models and decoders from MT models, and scripts support integrating released wav2vec2.0 and BERT components.Initialization is supported when model variable names match.Source: §2.2 · p. 3
• NeurST supports streaming-audio or streaming-text simultaneous translation and customizable validation, checkpoint tracking, and TensorBoard training metrics.Standard Transformer models can be served with TensorFlow Serving or optimized inference libraries such as lightseq.Source: §2.2 · p. 3

3 Speech Translation Benchmarks

• The benchmark section aims to provide strong, reproducible comparisons despite differences in model structures and hyperparameters across systems.The reported comparisons cover several benchmark speech translation corpora and other open-source codebases and studies.Source: §3 · p. 3

3.1 Datasets

• The benchmarks include libri-trans, a small EN→FR dataset with 236 hours of aligned English speech and French translations, and MuST-C, an English-to-eight-language corpus comprising at least 385 hours of audio.Libri-trans training uses the clean 100-hour portion plus augmented machine translation data; MuST-C uses dev and tst-COMMON for development and testing.Source: §3.1 · p. 3

3.2 Data Preprocessing

• Preprocessing extracts and normalizes speech features, then tokenizes and applies shared BPE subwords; the released datasets receive no additional alignment or cleaning.The text pipeline removes punctuation and lowercases source sentences while preserving target case and punctuation.Source: §3.2 · pp. 3–4

3.3 Benchmark Models

• The benchmark uses Transformer models; ASR/ST encode audio with CNN compression, and the ST encoder is initialized from ASR by default.ST also uses SpecAugment and a larger encoder than MT; the benchmark models are described as Transformer implementations.Source: §3.3 · pp. 4–5

3.4 Evaluation

• Evaluation reports WER for ASR and BLEU for MT/ST, with case-insensitive tokenized BLEU additionally reported for libri-trans comparisons.The evaluation averages the latest 10 checkpoints and uses beam width 4 without length penalty.Source: §3.4 · p. 5

3.5 Main Results

• 0.5 BLEU over ESPnet-ST on libri-trans; NeurST is also competitive with fairseq-ST and ESPnet-ST on MuST-C tst-COMMON.The reported libri-trans results also place end-to-end ST 0.4–0.5 BLEU above the cascade system; all main results are from single models.Source: Table 2 · p. 4, Table 3 · p. 4, §3.5 · p. 5

3.6 Ablation Study

• ASR encoder initialization yields reasonable or better BLEU, MT decoder initialization has relatively marginal impact, and SpecAugment consistently boosts ST models.The ablations are reported for libri-trans and MuST-C EN-DE; the cited tables identify their differing setups and BLEU measures.Source: §3.6 · pp. 5–6, Table 5 · p. 5, Table 6 · p. 5

3.7 Cascade versus End-to-End

• In the large-data setting, cascade ST is 1.7 BLEU ahead of end-to-end ST, reversing earlier libri-trans and MuST-C NL/PT results favoring end-to-end systems.The authors suggest self-supervised learning, knowledge distillation, and dataset construction as future ways to realize end-to-end models’ potential.Source: §3.7 · p. 6

• NeurST provides straightforward recipes for audio data preprocessing, training, and inference.Source: §4 · p. 6
• The authors report strong, reproducible benchmarks that can serve as reliable baselines for speech translation research.Source: §4 · p. 6
• The authors describe the toolkit as friendly to NLP researchers and say they will continue tracking advances using NeurST.Source: §4 · p. 6