Source-linked AI summary
The PyTorch-Kaldi Speech Recognition Toolkit
Mirco Ravanelli, Titouan Parcollet, Yoshua Bengio
TL;DR
ASR research benefits from separate mature toolkits: Kaldi offers an efficient recognition pipeline, while PyTorch offers flexible neural-network design. PyTorch-Kaldi bridges them with configurable hybrid DNN-HMM components and supports customized or combined architectures. Experiments across multiple datasets and tasks show that it can develop competitive state-of-the-art speech-recognition systems.
Problem
Kaldi and PyTorch provide complementary strengths, but a toolkit is needed to combine Kaldi’s speech-recognition pipeline with PyTorch’s flexible neural modeling.
Method
PyTorch-Kaldi implements PyTorch acoustic models alongside Kaldi feature extraction, alignment, decoding, and configurable combinations of models, features, and labels.
Results
PyTorch-Kaldi achieves competitive state-of-the-art results across several speech-recognition datasets and tasks.
Takeaways & Limitations
The publicly released toolkit makes flexible modern speech-recognizer development simpler while supporting customized acoustic models and complex ASR pipelines.
Abstract
from arXiv · showhide
The availability of open-source software is playing a remarkable role in the popularization of speech recognition and deep learning. Kaldi, for instance, is nowadays an established framework used to develop state-of-the-art speech recognizers. PyTorch is used to build neural networks with the Python language and has recently spawn tremendous interest within the machine learning community thanks to its simplicity and flexibility. The PyTorch-Kaldi project aims to bridge the gap between these popular toolkits, trying to inherit the efficiency of Kaldi and the flexibility of PyTorch. PyTorch-Kaldi is not only a simple interface between these software, but it embeds several useful features for developing modern speech recognizers. For instance, the code is specifically designed to naturally plug-in user-defined acoustic models. As an alternative, users can exploit several pre-implemented neural networks that can be customized using intuitive configuration files. PyTorch-Kaldi supports multiple feature and label streams as well as combinations of neural networks, enabling the use of complex neural architectures. The toolkit is publicly-released along with a rich documentation and is designed to properly work locally or on HPC clusters. Experiments, that are conducted on several datasets and tasks, show that PyTorch-Kaldi can effectively be used to develop modern state-of-the-art speech recognizers.
1. INTRODUCTION
ASR has advanced through deep learning, open datasets, challenges, and open-source toolkits, but Kaldi’s efficiency and PyTorch’s neural-network flexibility remain separate strengths. PyTorch-Kaldi bridges them for flexible, competitive speech-recognition development.
- Open-source software, public datasets, and speech-recognition challenges have helped popularize ASR and establish common evaluation frameworks.
- Kaldi efficiently implements state-of-the-art speech recognizers through C++ libraries and recipes covering popular speech corpora.
- PyTorch provides efficient GPU tensor computation, automatic gradients, and dynamically constructed computational graphs for flexible neural architectures.
- PyTorch-Kaldi combines PyTorch acoustic models with Kaldi feature extraction, alignment, and decoding for DNN-HMM speech recognition.
- The toolkit supports customized acoustic models, complex combinations of models, features, and labels, and configuration-based exploration of ASR hyperparameters.
- Experiments across several datasets and tasks show that PyTorch-Kaldi can support easily developed competitive state-of-the-art speech-recognition systems.
2. THE PYTORCH-KALDI PROJECT
PyTorch-Kaldi uses a Python-controlled pipeline that combines Kaldi’s speech-processing components with PyTorch neural models. It supports configurable data processing, multiple streams and labels, recurrent batching, custom models, pre-defined architectures, and Kaldi decoding.
- Unlike a Kaldi wrapper, PyTorch-Kaldi provides pre-implemented neural models while implementing hybrid DNN-HMM recognizers rather than primarily end-to-end recognition.
- The Python script run exp.py manages feature and label extraction, training, validation, decoding, and scoring across the ASR pipeline.
- INI configuration sections specify experiment settings, dataset features and labels, context windows, and chunking choices.
- Kaldi extracts features into binary archives, which Python utilities import and process through context composition, shuffling, normalization, and multiple feature streams.
- Forced-alignment phone-state labels support acoustic-model training, while multiple labels enable multitask learning and monophone regularization.
- 2.4. Chunk and Mini-batch Composition: The toolkit dynamically creates randomized chunks and network-specific minibatches, sorting recurrent sequences by length to reduce padding and support stable training.
- 2.5. DNN acoustic modeling: Users can add customized models through a class defining initialization and forward computation, or configure built-in MLP, CNN, RNN, LSTM, GRU, Light GRU, twin-regularized RNN, and SincNet models.
- Kaldi’s decoder combines normalized neural posteriors with n-gram language probabilities, then computes WER using NIST SCTK.
3. EXPERIMENTAL SETUP
The experiments evaluate PyTorch-Kaldi on phoneme and speech-recognition tasks spanning clean, distant-talking, noisy, and large-vocabulary settings. They vary corpora, acoustic features, neural initialization, and regularization under Kaldi-based protocols.
- 3.1. Corpora and Tasks: The TIMIT experiments use the standard phoneme-recognition task aligned with the Kaldi s5 recipe.
- 3.1. Corpora and Tasks: DIRHA-English evaluates distant-talking recognition using contaminated WSJ-5k training data and real domestic-environment test speech.
- 3.1. Corpora and Tasks: CHiME 4 evaluates single-channel recognition on noisy speech recorded across bus, cafe, pedestrian-area, and street-junction environments.
- 3.1. Corpora and Tasks: LibriSpeech experiments use 100 hours of training data, dev-clean for tuning, and test-clean decoded with the Kaldi s5 fglarge graph.
- The study compares 39-dimensional MFCCs, 40-dimensional log-mel filter-bank features, and 40-dimensional fMLLR features computed using 25-ms windows with 10-ms overlap.
- Feed-forward models use Glorot initialization, recurrent weights use orthogonal initialization, and recurrent dropout provides regularization.
4. BASELINES
Experiments across TIMIT, DIRHA, CHiME, and LibriSpeech show that PyTorch-Kaldi supports competitive speech-recognition systems through varied models, features, and architectures.
- TIMIT baselines: PER=14.2% is achieved by Li-GRU on TIMIT, outperforming standard MLPs and benefiting from a single gate that saves 33% of GRU computations.fMLLR features outperform MFCCs and FBANKs, while recurrent models outperform the standard MLP.
- Improvement techniques: Progressive sequence-length training, recurrent dropout, batch normalization, and monophone regularization are evaluated as successive performance-improvement techniques.Progressive truncation begins at approximately one second and doubles the maximum sequence duration each epoch; recurrent dropout uses one mask across time steps.
- TIMIT baselines: PER=13.8% is obtained by combining feed-forward and recurrent models with concatenated features, reported as the best-published TIMIT test-set performance.
- Alternative features: Raw-waveform CNNs perform similarly to FBANK-based CNNs, while SincNet improves performance and demonstrates effectiveness in speech recognition.These experiments use user-provided features rather than relying exclusively on features computed with Kaldi.
- Cross-dataset validation: Li-GRU consistently performs better across DIRHA, CHiME, and LibriSpeech, including noisy conditions and a 100-hour LibriSpeech subset.DIRHA reaches WER=23.9% on the single-microphone task, while the LibriSpeech result outperforms the p-norm Kaldi baseline with WER = 6.5%.
5. CONCLUSIONS
PyTorch-Kaldi bridges Kaldi and PyTorch by simplifying flexible ASR development and supporting customized, composable pipelines. Experiments show that it can achieve state-of-the-art results on several speech-recognition tasks and datasets.
- PyTorch-Kaldi bridges Kaldi and PyTorch while simplifying ASR development through user-defined acoustic models.
- The toolkit supports combinations of neural architectures, features, and labels, enabling complex ASR pipelines.
- Experiments confirm that PyTorch-Kaldi can achieve state-of-the-art results on several popular speech-recognition tasks and datasets.
- PyTorch-Kaldi is publicly available with detailed documentation, and its developers invite contributors to maintain and expand it.