Source-linked AI summary
Fast and Accurate Recurrent Neural Network Acoustic Models for Speech Recognition
Haşim Sak, Andrew Senior, Kanishka Rao, Françoise Beaufays
TL;DR
The paper addresses how to improve LSTM RNN acoustic models for large-vocabulary speech recognition beyond prior DNN and CTC-based systems. It combines frame stacking and reduced frame rates with sequence-trained CTC and context-dependent phone models, and explores direct word outputs. The resulting CTC context-dependent phone models outperform conventional LSTM-hybrid models, while word models achieve reasonable accuracy on medium vocabularies.
Problem
The paper seeks more accurate and faster LSTM RNN acoustic models for large-vocabulary speech recognition, building beyond prior CTC phone models and conventional LSTM-hybrid systems.
Method
The paper applies frame stacking and reduced frame rates, trains CTC models with sMBR sequence discrimination and context-dependent phones, and evaluates direct word-level acoustic models.
Results
CTC models with context-dependent phones outperform conventional LSTM RNN models by 8% relative in recognition accuracy.
Takeaways & Limitations
Longer-term, lower-rate features and context-dependent units improve CTC acoustic models, while direct word models achieve reasonable accuracy on medium-vocabulary recognition.
Takeaways & Limitations
CTC without a blank label learns an arbitrary alignment because recurrent models can delay outputs instead of relying only on local temporal information.
Abstract
from arXiv · showhide
We have recently shown that deep Long Short-Term Memory (LSTM) recurrent neural networks (RNNs) outperform feed forward deep neural networks (DNNs) as acoustic models for speech recognition. More recently, we have shown that the performance of sequence trained context dependent (CD) hidden Markov model (HMM) acoustic models using such LSTM RNNs can be equaled by sequence trained phone models initialized with connectionist temporal classification (CTC). In this paper, we present techniques that further improve performance of LSTM RNN acoustic models for large vocabulary speech recognition. We show that frame stacking and reduced frame rate lead to more accurate models and faster decoding. CD phone modeling leads to further improvements. We also present initial results for LSTM RNN models outputting words directly.
1. Introduction
Recent work established that recurrent neural networks can outperform feed-forward networks for large-scale speech recognition. This paper investigates techniques for further improving LSTM RNN acoustic models, including frame processing, context-dependent phones, and direct word outputs.
- RNNs have recently displaced GMMs as state-of-the-art acoustic models in speech recognition.
- The paper further investigates sMBR-trained CTC models for acoustic speech recognition.
- CTC models with appropriate features and context-dependent phone models outperform conventional LSTM RNN models by 8% relative in recognition accuracy.
- The study also evaluates LSTM RNN acoustic models that output whole words directly.
2. RNN Acoustic Modeling Techniques
The paper combines deep LSTM architectures, CTC alignment learning, sequence training, frame subsampling, context-dependent phones, and word-level modeling to improve speech-recognition acoustic models.
- RNN Acoustic Modeling Techniques: Deep LSTM RNNs process sequences unidirectionally for low latency or bidirectionally when full-sequence context is available.
- CTC Training: CTC learns unknown input-label alignments using a blank symbol and forward-backward optimization over valid label sequences.
- Sequence Discriminative Training: After sequence discriminative training, CTC and conventional models differ only in whether they use the blank symbol.
- Acoustic Features: Frame stacking and subsampling let the model process the full signal while computing acoustic outputs every 30ms, reducing computation and decoding time.
- Context-Dependent Phones: Context-dependent phones are introduced because context-dependent states outperform context-independent models in conventional speech recognition systems.
- Word Acoustic Models: LSTM memory and CTC alignment learning enable acoustic models with longer-duration units, including whole words across 7,000-to-90,000-word vocabularies.
3. Experiments
Experiments evaluate LSTM RNN acoustic models on voice-search speech recognition using conventional and CTC training, sequence-discriminative training, posterior analyses, and direct word prediction. CTC CD phone models and bidirectional architectures improve WER, while direct CTC word models provide initial results across vocabulary sizes.
- Experimental setup: The voice-search experiments use 3 million training utterances and 28,000 distorted test utterances, with a pruned 5-gram language model for evaluation.Training data are synthetically distorted into 20 variants per utterance, while test utterances use held-out noise samples.
- Results: CTC CD phone models improve substantially over CTC CI phone models, by about 8% for unidirectional and 3.5% for bidirectional models.Bidirectional models improve over unidirectional models by about 10% for CD-state and CI-phone models, but only 5% for CTC CD-phone models.
- Sequence training: sMBR sequence training improves CE- and CTC-initialized models by about 10% relative, with CTC CD phone models achieving the best results.The best models outperform the second-best model by about 8% for unidirectional and 4% for bidirectional systems.
- Posterior analysis: CTC posterior spikes differ from fixed DNN alignments, unidirectional models delay outputs by about 300 milliseconds, and sequence training changes posteriors without shifting spike positions.Bidirectional models make better predictions, but the models are not good at modeling silence labels.
- Alignment analysis: Without a blank label, a unidirectional CTC CD-phone model learns an arbitrary alignment because recurrent memory permits delayed outputs instead of local decisions.The resulting alignment can repeat labels unevenly according to input-dependent certainty.
- Word models: Direct CTC word models are evaluated with 7,000-, 25,000-, and 90,000-word vocabularies, using greedy edit-distance WER without a language model or decoding.For the 90k vocabulary, the bidirectional model has 25% lower WER than the unidirectional model; OOV words affect the 7k analysis.
4. Conclusions
Longer-term features processed at lower frame rates stabilize CTC training while reducing computation. Sequence-trained blank-symbol models improve over prior acoustic models, context-dependent phones improve them further, and word-level models achieve reasonable medium-vocabulary accuracy without a language model.
- Longer-term feature representations processed at lower frame rates stabilize CTC convergence and considerably reduce computation.
- Sequence-trained models with these representations perform better than previous acoustic models.
- Context-dependent phonetic units further improve blank-symbol acoustic models beyond conventional sequence-trained LSTM-hybrid models.
- Word-level acoustic models achieve reasonable accuracy on medium-vocabulary speech recognition without using a language model.