Source-linked AI summary
RWTH ASR Systems for LibriSpeech: Hybrid vs Attention -- w/o Data Augmentation
Christoph Lüscher, Eugen Beck, Kazuki Irie, Markus Kitza, Wilfried Michel, Albert Zeyer, Ralf Schlüter, Hermann Ney
TL;DR
The paper asks how conventional hybrid DNN/HMM and attention-based encoder-decoder systems compare on LibriSpeech. It develops and evaluates both architectures with neural acoustic and language models, finding that the hybrid system performs better, especially across the full and reduced-data settings.
Problem
The paper examines the relative performance of hybrid DNN/HMM and attention-based encoder-decoder architectures for LibriSpeech ASR.
Method
The authors develop both architectures using bi-directional LSTM acoustic modeling or encoding, LSTM and Transformer language models, and incremental training procedures.
Results
15% relative on clean and 40% relative on other test sets: the hybrid DNN/HMM system outperforms the attention-based system on LibriSpeech 960h in WER.
Takeaways & Limitations
The hybrid system outperforms the attention-based system on the 960h task, while the attention model achieves competitive end-to-end results without data augmentation.
Takeaways & Limitations
The evaluation does not use a normalization strategy for errors involving contractions and American versus British English spelling.
Abstract
from arXiv · showhide
We present state-of-the-art automatic speech recognition (ASR) systems employing a standard hybrid DNN/HMM architecture compared to an attention-based encoder-decoder design for the LibriSpeech task. Detailed descriptions of the system development, including model design, pretraining schemes, training schedules, and optimization approaches are provided for both system architectures. Both hybrid DNN/HMM and attention-based systems employ bi-directional LSTMs for acoustic modeling/encoding. For language modeling, we employ both LSTM and Transformer based architectures. All our systems are built using RWTHs open-source toolkits RASR and RETURNN. To the best knowledge of the authors, the results obtained when training on the full LibriSpeech training set, are the best published currently, both for the hybrid DNN/HMM and the attention-based systems. Our single hybrid system even outperforms previous results obtained from combining eight single systems. Our comparison shows that on the LibriSpeech 960h task, the hybrid DNN/HMM system outperforms the attention-based system by 15% relative on the clean and 40% relative on the other test sets in terms of word error rate. Moreover, experiments on a reduced 100h-subset of the LibriSpeech training corpus even show a more pronounced margin between the hybrid DNN/HMM and attention-based architectures.
1. Introduction
The paper compares conventional hybrid DNN/HMM recognition with attention-based encoder-decoder systems on LibriSpeech. It also examines language-modeling improvements for both architectures.
- Hybrid DNN/HMM systems were established as standard state-of-the-art ASR architectures, while attention-based encoder-decoder models represented a prominent end-to-end alternative.
- Previous LibriSpeech hybrid results included systems using GMM/HMM, densely connected neural networks, LF-MMI, recurrent neural-network language-model rescoring, and system combination.
- Prior end-to-end LibriSpeech work explored raw-waveform models, attention-based pretraining, edit-distance optimization, target units, and data augmentation.
- The paper compares phone-level hybrid DNN/HMM recognition with word- or subword-level attention-based encoder-decoder recognition and applies word- and subword-level neural language models.
2. Hybrid system
The hybrid system is developed incrementally from a GMM/HMM alignment model into a bi-directional LSTM acoustic model with sequence training and neural language-model rescoring.
- GMM/HMM system: The GMM/HMM pipeline progresses from linear and nonlinear alignment through triphone modeling, VTLN, SAT, and realignment.State-tied triphones use CART-based tied states and LDA-transformed contextual MFCC features.
- Hybrid DNN/HMM system: The hybrid acoustic model uses six bi-directional LSTM layers with 1000 forward and 1000 backward units per layer.MFCC, energy, triphone, and Gammatone features are supplied alongside GMM/HMM-generated alignments.
- Hybrid DNN/HMM system: Sequence discriminative training continues hybrid-model training using a lattice-based sMBR criterion.The hybrid model generates training lattices, and subsequent training uses a lower learning rate with cross-entropy smoothing and early stopping.
- Language modeling: Hybrid decoding uses either the official 4-gram language model or an LSTM language model, followed by optional Transformer rescoring.The LSTM language model is trained on text-only and audio-transcription text datasets.
3. Encoder-Decoder-Attention system
The attention-based encoder-decoder system operates on BPE subwords and uses staged curriculum learning, extended pretraining, longer training, and neural-language-model shallow fusion.
- The encoder-decoder attention model recognizes sub-word units represented with byte-pair encoding from 40-dimensional MFCC features.
- Training schedule: Training uses a staged curriculum that progressively increases transcription complexity and eventually includes the full training data.The stages use train-clean subsets with maximum mean transcription lengths of 50 and 75 characters before unrestricted training.
- Evaluation reference: Table 1 reports GMM/HMM and hybrid DNN/HMM results using 12k CART labels and the official 4-gram language model.
- Pretraining and optimization: Extended pretraining and repeated learning-rate resets continue optimization until convergence, totaling 27.4 passes over the data.The reported continuation stages span 12.5, 6.6, and 8.3 passes.
- Language modeling: BPE-level LSTM and Transformer language models are applied through shallow fusion to improve end-to-end recognition.The LSTM model has four recurrent layers, while the Transformer model uses 24 layers and eight attention heads.
4. Experimental setup
Both architectures are evaluated on LibriSpeech using the full 960-hour training set and a 100-hour subset, with results reported across clean and other development and test sets.
- Training data: The hybrid DNN/HMM and attention-based encoder-decoder systems are trained on LibriSpeech’s 960-hour corpus, with an additional 100-hour comparison.Unless otherwise specified, training uses the full 960-hour set.
- Toolkits: The hybrid system uses RASR for alignment and decoding and RETURNN for neural-network training, while the encoder-decoder system uses RETURNN for training and decoding.Both toolkits are open-source, and the training and recognition configuration files are publicly available.
- Evaluation: Evaluation covers dev-clean, dev-other, test-clean, and test-other from the LibriSpeech corpus.Clean sets have higher audio and transcription quality than other sets.
5. Experimental results
Experiments compare hybrid DNN/HMM and attention-based encoder-decoder systems across model-development choices, training-set sizes, and published LibriSpeech results. The hybrid system generally performs better, while larger training data narrows the baseline gap and normalization remains unaddressed for some transcription variants.
- Sequence discriminative training further improves the hybrid acoustic model after introducing the hybrid DNN/HMM architecture.
- 12k CART labels achieve the best performance among the evaluated hybrid DNN/HMM configurations, whereas 9k labels perform worst.
- The hybrid DNN/HMM model consistently outperforms the attention-based encoder-decoder baseline, but the gap shrinks substantially with the train-960 corpus.
- The best reported hybrid system reaches 2.3% WER on test-clean and 5.0% on test-other after Transformer LM rescoring.
- The best attention-based system reaches 3.2% WER on test-clean and 9.9% on test-other with a Transformer LM.
- Some residual WER reflects contractions and spelling variants that were not handled by a normalization strategy.
6. Conclusions
The paper develops and compares hybrid DNN/HMM and attention-based encoder-decoder ASR systems for LibriSpeech. Both achieve competitive or state-of-the-art results, but the hybrid system has a substantial relative WER advantage, especially on the full task and reduced-data setting.
- The paper presents a hybrid DNN/HMM system and an attention-based encoder-decoder system for LibriSpeech.
- The hybrid system relies on a large acoustic model, sequence discriminative training, and an LSTM language model, while the attention system uses extended pretraining and curriculum learning.
- The attention-based system achieves state-of-the-art performance among end-to-end systems without data augmentation on LibriSpeech 960h.
- The hybrid system outperforms the attention-based system by 15% relative on clean and 40% relative on other test sets for LibriSpeech 960h.
- The performance margin is more pronounced on the reduced 100h LibriSpeech subset, and the authors report their full-data results as the best published for both architectures.