Source-linked AI summary
fairseq S2T: Fast Speech-to-Text Modeling with fairseq
Changhan Wang, Yun Tang, Xutai Ma, Anne Wu, Sravya Popuri, Dmytro Okhonko, Juan Pino
TL;DR
Speech-to-text research needs scalable tooling that connects recognition, translation, machine translation, and language modeling. FAIRSEQ S2T provides integrated workflows, model families, training recipes, and evaluation support, with competitive ASR and speech-translation results across benchmarks and settings.
Problem
Connected ASR, ST, MT, and LM workflows, together with large-scale unlabeled speech, require an integrated and scalable S2T toolkit.
Method
FAIRSEQ S2T extends FAIRSEQ with end-to-end preprocessing, training, inference, RNN, Transformer, and Conformer models, plus MT and LM integration.
Results
FAIRSEQ S2T matches state-of-the-art ASR performance with its Conformer implementation and achieves competitive or improved results across MuST-C and CoVoST 2 settings.
Takeaways & Limitations
The toolkit provides scalable, extensible, and reproducible workflows for speech recognition and speech translation across architectures, languages, and latency regimes.
Abstract
from arXiv · showhide
We introduce fairseq S2T, a fairseq extension for speech-to-text (S2T) modeling tasks such as end-to-end speech recognition and speech-to-text translation. It follows fairseq's careful design for scalability and extensibility. We provide end-to-end workflows from data pre-processing, model training to offline (online) inference. We implement state-of-the-art RNN-based, Transformer-based as well as Conformer-based models and open-source detailed training recipes. Fairseq's machine translation models and language models can be seamlessly integrated into S2T workflows for multi-task learning or transfer learning. Fairseq S2T documentation and examples are available at https://github.com/pytorch/fairseq/tree/master/examples/speech_to_text.
1 Introduction
FAIRSEQ S2T addresses the need for scalable, integrated tooling across speech recognition, speech translation, machine translation, and language modeling. It provides reproducible workflows and supports multiple model families and learning paradigms.
- FAIRSEQ S2T is a FAIRSEQ extension for end-to-end automatic speech recognition and speech-to-text translation.
- The toolkit provides workflows spanning data preprocessing, model training, and offline or online inference.
- It implements RNN, Transformer, and Conformer models and provides open-source training recipes.
- FAIRSEQ machine translation models and language models integrate into S2T workflows for multi-task learning and transfer learning.
- The toolkit emphasizes integration, scalability, and reproducibility compared with counterpart toolkits.
2 Features
FAIRSEQ S2T extends FAIRSEQ with diverse speech-to-text architectures, preprocessing and configuration interfaces, scalable computation, and evaluation tools. It also supports simultaneous translation and integration with FAIRSEQ models.
- S2T extension: FAIRSEQ S2T supports attention-based RNN, Transformer, and Conformer models for ASR and ST, plus CTC for ASR.
- S2T extension: Simultaneous speech translation includes monotonic attention, wait-k, monotonic infinite lookback, and monotonic multihead policies.
- Data Pre-Processing: Speech preprocessing supports automatic feature extraction, packed audio or feature archives, CMVN, speed perturbation, SpecAugment, and user-defined transforms.
- Data Configuration: Manifest files store audio paths and target texts, while YAML configuration files define tokenization, vocabulary, and other data settings.
- Computation: PyTorch-based computation provides efficient batching, mixed precision, multi-GPU, and multi-machine training for large-scale experiments.
- Evaluation Metrics: Evaluation covers WER, BLEU, chrF, and simultaneous translation metrics through SIMULEVAL, while VizSeq supports sequence-level visual error analysis.
3 Experiments
Experiments evaluate FAIRSEQ S2T on LibriSpeech, MuST-C, and CoVoST 2 using standardized configurations and established speech recognition and translation metrics. Results are competitive across architectures, languages, and latency settings.
- Benchmarks: Experiments use LibriSpeech for English ASR and MuST-C and CoVoST 2 for multilingual speech translation.LibriSpeech contains 1,000 hours of English audiobook speech; MuST-C covers up to around 500 hours and eight European languages.
- Experimental Setup: The setup uses 80-channel log mel-filter banks, utterance-level CMVN, SpecAugment, checkpoint averaging, beam size 5, and task-specific vocabularies and metrics.
- Speech Recognition (ASR): FAIRSEQ S2T achieves competitive WER with RNN and Transformer models using default hyperparameters and learning-rate schedules without task-specific tuning.
- Speech Recognition (ASR): Its Conformer-based wav2vec 2.0 implementation matches the state of the art on LibriSpeech.
- MuST-C: On MuST-C, bilingual models achieve comparative results to prior Transformer approaches, while the multilingual model outperforms all bilingual models with large margins.
- CoVoST 2: On CoVoST 2, Transformer models outperform RNN models across evaluated directions, while multilingual models perform reasonably well across more than 15 directions.Self-supervised speech features bring consistent gains and transfer across languages in low-resource directions.
4 Conclusion
FAIRSEQ S2T extends FAIRSEQ for speech-to-text modeling with scalable, extensible design and end-to-end workflows. It also integrates FAIRSEQ’s machine translation and language models into S2T workflows.
- FAIRSEQ S2T extends FAIRSEQ for speech-to-text modeling tasks including speech recognition and speech translation.
- The extension provides end-to-end workflows spanning data preprocessing, model training, and offline or online inference.
- FAIRSEQ S2T is designed for scalability and extensibility.
- FAIRSEQ’s machine translation and language models integrate into S2T workflows to improve S2T model performance.