Source-linked AI summary
EESEN: End-to-End Speech Recognition using Deep RNN Models and WFST-based Decoding
Yajie Miao, Mohammad Gowayyed, Florian Metze
TL;DR
Eesen addresses the complexity of building ASR systems by using a single CTC-trained RNN acoustic model and WFST-based decoding. It achieves WERs comparable to hybrid HMM/DNN baselines while speeding up decoding, though it cannot leverage speaker-adapted front-ends.
Problem
Building state-of-the-art ASR systems still requires substantial resources, multiple training stages, pre-generated frame labels, and extensive expertise.
Method
Eesen trains deep RNN acoustic models with CTC on context-independent labels and composes CTC labels, lexicons, and language models in WFST search graphs.
Results
Eesen achieves WERs comparable to strong hybrid HMM/DNN baselines and speeds decoding by reducing the number of modeling states.
Takeaways & Limitations
Eesen provides an open-source framework and shared benchmark platform for simplifying end-to-end ASR research and development.
Takeaways & Limitations
Because GMMs are removed, Eesen cannot leverage speaker-adapted front-ends and requires new speaker-adaptation or adaptive-training techniques.
Abstract
from arXiv · showhide
The performance of automatic speech recognition (ASR) has improved tremendously due to the application of deep neural networks (DNNs). Despite this progress, building a new ASR system remains a challenging task, requiring various resources, multiple training stages and significant expertise. This paper presents our Eesen framework which drastically simplifies the existing pipeline to build state-of-the-art ASR systems. Acoustic modeling in Eesen involves learning a single recurrent neural network (RNN) predicting context-independent targets (phonemes or characters). To remove the need for pre-generated frame labels, we adopt the connectionist temporal classification (CTC) objective function to infer the alignments between speech and label sequences. A distinctive feature of Eesen is a generalized decoding approach based on weighted finite-state transducers (WFSTs), which enables the efficient incorporation of lexicons and language models into CTC decoding. Experiments show that compared with the standard hybrid DNN systems, Eesen achieves comparable word error rates (WERs), while at the same time speeding up decoding significantly.
1. INTRODUCTION
Eesen simplifies end-to-end ASR development by replacing a multistage hybrid pipeline with a single CTC-trained RNN acoustic model and WFST-based decoding. It incorporates lexicons and language models while achieving WERs comparable to hybrid HMM/DNN systems.
- Hybrid ASR development remains complex because it requires substantial resources, multiple training stages, frame-level labels, and expert tuning.GMM-based preparation and numerous hyper-parameter choices complicate construction, especially when resources are unavailable.
- Eesen addresses a major end-to-end ASR obstacle by enabling efficient incorporation of word-level language models into CTC decoding.Earlier work had incorporated lexicons, but efficient word-level language-model integration remained unresolved.
- Eesen uses deep RNNs with CTC to learn a single acoustic model from speech and context-independent label sequences without pre-generated frame labels.CTC automatically infers alignments between speech frames and phoneme or character sequences.
- Eesen’s generalized WFST decoder encodes CTC labels, lexicons, and language models into one composed search graph.The representation handles the CTC blank label and supports beam search during decoding.
- On the WSJ benchmark, Eesen achieves WERs comparable to strong hybrid HMM/DNN baselines while outperforming existing end-to-end ASR pipelines.The framework also reduces decoding time and memory usage through context-independent modeling targets.
2. THE EESEN FRAMEWORK: MODEL TRAINING
Eesen trains deep bidirectional LSTM RNNs with CTC, replacing frame-level labels with sequence-level alignment learning. Its GPU implementation parallelizes utterance and forward-backward computations for training efficiency.
- Deep Bidirectional Recurrent Neural Networks: Eesen acoustic models stack multiple bidirectional recurrent layers, passing concatenated forward and backward outputs between layers.The architecture uses deep bidirectional RNNs, with each layer receiving both directional hidden representations from the preceding layer.
- Deep Bidirectional Recurrent Neural Networks: LSTM units address vanishing gradients by storing temporal states in memory cells and controlling information flow with multiplicative gates.The paper uses a purely LSTM-based acoustic architecture, while noting that other structures could be combined with LSTMs.
- Training with Connectionist Temporal Classification: CTC automatically learns alignments between speech frames and phoneme or character sequences without frame-level labels or cross-entropy training.CTC adds a blank label and maximizes the log-likelihood of the label sequence given the input utterance.
- Training with Connectionist Temporal Classification: CTC paths permit blanks and repeated labels, allowing multiple frame-level paths to map to the same target label sequence.The target likelihood is formed by summing probabilities over the corresponding CTC paths.
- Training with Connectionist Temporal Classification: A trellis and forward-backward algorithm evaluate CTC likelihoods efficiently, while derivatives are propagated through the softmax layer into the RNN.The augmented label sequence inserts blank symbols between labels and at both ends to support the dynamic program.
- Implementation: GPU training processes multiple utterances and independent forward-backward elements in parallel, using padding masks to exclude artificial frames from gradients.This replaces single-frame matrix-vector operations with matrix-matrix operations and parallelizes computations at each frame.
3. THE EESEN FRAMEWORK: DECODING
Eesen decodes CTC outputs by representing labels, lexicons, and language models as WFSTs, composing them into a search graph that maps frame-level labels to words. The framework also addresses CTC-specific posterior normalization, whose naive priors are dominated by blank labels.
- WFST decoding: WFST decoding represents CTC labels, lexicons, and language models separately, then fuses them into one comprehensive search graph.The approach uses optimized FST libraries to compose the individual components efficiently.
- Grammar WFST: The grammar WFST encodes permissible word sequences, with arc weights representing language-model probabilities conditioned on preceding words.The example grammar permits the sentences “how are you” and “how is it.”
- Lexicon WFST: The lexicon WFST maps phoneme or character sequences to words; character spelling lexicons can more easily incorporate out-of-vocabulary words than phoneme lexicons.Phoneme-lexicon expansion may require grapheme-to-phoneme rules or models and can be subject to errors.
- Token WFST: The token WFST maps frame-level CTC sequences to one lexicon unit while allowing blank labels and repetitions of non-blank labels.For example, multiple five-frame sequences involving “IH” can map to the singleton phoneme “IH.”
- Search graph: Composition, determinization, and minimization produce a compressed search graph that maps frame-level CTC labels to word sequences.The lexicon and grammar are composed first, then the result is composed with the token WFST.
- Posterior normalization: Naive CTC label-prior estimation performs poorly because peaky outputs assign most frames to blank, dominating the prior counts.The proposed alternative estimates priors by counting augmented training label sequences containing blanks between and around labels.
4. EXPERIMENTS
Experiments on WSJ evaluate Eesen with phoneme- and character-based systems, comparing recognition accuracy and decoding efficiency against hybrid HMM/DNN systems and prior end-to-end work.
- 4.2. Phoneme-based Systems: On WSJ eval92, the phoneme-based Eesen system achieves 7.87% WER using both a lexicon and language model.The model uses four bidirectional LSTM layers, each with 320 forward and backward memory cells.
- 4.2. Phoneme-based Systems: Without the language model, phoneme-based decoding reaches 26.92% WER, showing the effect of incorporating language models during WFST decoding.Lexicon-only decoding behaves similarly to beam search in prior work.
- 4.2. Phoneme-based Systems: The phoneme-based Eesen system remains behind the hybrid HMM/DNN system in recognition performance on the reported WSJ comparison.The compared Eesen and DNN models contain 8.5 and 9.2 million parameters, respectively.
- 4.2. Phoneme-based Systems: Eesen decoding is 3.2× faster than HMM/DNN decoding, while its TLG graph is significantly smaller than the HCLG graph.The speedup is attributed to reducing modeled states from thousands of senones to tens of phonemes or characters, also saving graph storage space.
- 4.3. Character-based Systems: With the standard language model, the character-based system achieves 9.07% WER.The character system models 59 labels and uses the same RNN architecture as the phoneme-based system.
- 4.3. Character-based Systems: An expanded vocabulary and retrained language model reduce the character-based system’s WER to 7.34%, while Eesen outperforms prior end-to-end systems on the same test set.The comparison notes that Eesen’s results come from a completely end-to-end pipeline without GMM or hybrid DNN intervention.
5. CONCLUSIONS AND FUTURE WORK
Eesen simplifies end-to-end ASR development through single-step RNN training, CTC, and WFST-based decoding, while providing an open-source benchmark platform. Future work targets improved WERs, broader speech conditions, and speaker adaptation.
- Conclusions: Eesen uses deep RNNs with CTC to train acoustic models in a single step, reducing ASR system development complexity.The framework uses WFST-based decoding to incorporate lexicons and language models efficiently.
- Conclusions: WFST-based decoding enables efficient and effective incorporation of lexicons and language models.
- Conclusions: Eesen’s open-source property allows it to serve as a shared benchmark platform for end-to-end ASR research.
- Future Work: Future work aims to improve Eesen WERs through advanced learning techniques and alternative decoding approaches.The paper also proposes applying Eesen to various languages and noisy or far-field speech.
- Future Work: Removing GMMs prevents Eesen’s acoustic modeling from leveraging speaker-adapted front-ends, motivating new speaker adaptation and adaptive training techniques.