Source-linked AI summary
Neural Speech Recognizer: Acoustic-to-Word LSTM Model for Large Vocabulary Speech Recognition
Hagen Soltau, Hank Liao, Hasim Sak
TL;DR
Conventional speech recognition typically relies on complex context-dependent sub-word models, while direct word modeling has been constrained by data sparsity. The paper trains a 100K-word bidirectional LSTM with CTC on 125,000 hours of semi-supervised data, producing an all-neural recognizer without a pronunciation lexicon or decoder. It achieves a 13.4% word error rate and outperforms a conventional context-dependent phone-based system on YouTube transcription.
Problem
Direct whole-word acoustic modeling was limited by data sparsity, while conventional systems relied on complex clustered context-dependent sub-word units.
Method
The authors train a deep bidirectional LSTM with CTC to predict a written vocabulary of 100K words using 125,000 hours of semi-supervised caption-filtered acoustic data.
Results
13.4% word error rate: the final CTC word model performs better than a well-trained conventional context-dependent phone-based system on YouTube video transcription.
Takeaways & Limitations
CTC word models can provide competitive large-vocabulary speech recognition with a greatly simplified all-neural architecture that avoids a pronunciation lexicon and decoder.
Abstract
from arXiv · showhide
We present results that show it is possible to build a competitive, greatly simplified, large vocabulary continuous speech recognition system with whole words as acoustic units. We model the output vocabulary of about 100,000 words directly using deep bi-directional LSTM RNNs with CTC loss. The model is trained on 125,000 hours of semi-supervised acoustic training data, which enables us to alleviate the data sparsity problem for word models. We show that the CTC word models work very well as an end-to-end all-neural speech recognition model without the use of traditional context-dependent sub-word phone units that require a pronunciation lexicon, and without any language model removing the need to decode. We demonstrate that the CTC word models perform better than a strong, more complex, state-of-the-art baseline with sub-word units.
1 Introduction
The work revisits whole-word acoustic modeling as a simpler alternative to conventional clustered sub-word systems. Large semi-supervised datasets and CTC-enabled bi-directional LSTMs make large-vocabulary word recognition feasible despite earlier data-sparsity concerns.
- Motivation: Conventional recognizers predominantly use clustered context-dependent sub-word units because earlier datasets were too small for effective word models.The authors describe this historical choice as potentially sub-optimal with today’s larger datasets.
- Motivation: 125,000 hours of semi-supervised data help alleviate the sparsity problem that previously limited direct word modeling.The data come from user-uploaded video captions filtered against ASR transcripts.
- Motivation: CTC alignment combined with LSTM memorization capacity enables neural networks to recognize whole words directly.CTC learns alignments between acoustic inputs and label sequences, while LSTMs provide sequence-modeling capacity.
- Related work: Earlier encoder-decoder and grapheme-based end-to-end systems remained behind conventional phone-based baselines on large-vocabulary recognition.The cited encoder-decoder work used dynamic attention, while grapheme systems achieved good but not fully comparable results.
2 Neural Speech Recognizer
The Neural Speech Recognizer is a deep bidirectional LSTM trained with CTC to predict word posteriors directly. Its design uses whole-word vocabularies, alignment lattices, and a simple posterior-to-word-sequence output path without beam-search decoding.
- Architecture: The NSR stacks multiple bidirectional LSTM layers, with forward and backward LSTMs at each depth connected to both preceding directions.Bidirectionality is chosen for offline recognition because it provides better accuracy in the described application.
- CTC training: CTC trains the network by maximizing the total probability of valid labelings that preserve target order while allowing repetitions and blank labels.The probability is computed with a forward-backward algorithm over alignment paths.
- Word outputs: The final softmax predicts word posteriors over a vocabulary-sized output layer, using written or spoken vocabularies mapped through a verbalization FST.Mapping written transcripts to spoken forms reduces sparsity and limits label ambiguity in spoken-vocabulary experiments.
- Decoding: The model can produce a recognized word sequence directly from its CTC outputs without beam-search decoding or a language model.Written-domain output may additionally be obtained by constructing and rescoring a word lattice with a written-domain language model.
- Training: Distributed asynchronous stochastic gradient descent trains the large models across many machines.The implementation also uses phone-model initialization, activation and gradient clipping, and a pipelined multi-layer LSTM kernel.
3 Experimental Setup
The experiments use large-scale YouTube data and compare conventional bidirectional-LSTM phone models with CTC word models. The comparisons examine training-data scale, vocabulary choices, CTC versus CE training, and recognition with or without language-model decoding.
- Data and evaluation: 125,000 hours of semi-supervised training data were mined from user-uploaded captions to support large neural speech-recognition models.The data were obtained using “islands of confidence” filtering, which selects suitable audio segments from captioned videos.
- Conventional context-dependent phone models: 29.0% WER was obtained with a 650-hour 3-state CD-triphone HMM, improving to 24.0% with sequence-level state-MBR training and adapted decoding.Using 5,000 hours reduced the same model’s error rate to 21.2%.
- Vocabulary: The experiments evaluated spoken and written word vocabularies of 82,473 and 97,827 words, with OOV rates of 0.63% and 0.7%, respectively.The baseline used a 500,000-word vocabulary with a 0.24% OOV rate.
- CTC word models: 18.7% error rate was achieved by the CTC word model trained on 50,000 hours, compared with 23.1% for the corresponding CE word model.The authors attribute the difference to the difficulty of predicting longer word units frame by frame with CE.
- No-decoding recognition: 13.9% WER was achieved by the written CTC word model without language-model decoding, versus 14.2% for the decoded conventional CD-phone model.The spoken CTC word model achieved 14.8% WER under the same no-LM, no-decoding setup.
- Language-model rescoring: 0.5% improvement reduced the written word model’s WER to 13.4% after lattice rescoring with a language model.This indicates a relatively small language-model contribution to the system’s accuracy.
4 Conclusions
The paper presents an end-to-end all-neural recognizer that predicts words with a bidirectional LSTM and CTC loss. Trained on 125,000 hours of public-caption data, it achieves 13.4% WER on a difficult YouTube transcription task while omitting a pronunciation lexicon and decoder.
- The Neural Speech Recognizer is an end-to-end large-vocabulary speech recognizer without a pronunciation lexicon or decoder.
- The system uses a bidirectional LSTM RNN trained with CTC loss on 125,000 hours of mined public-caption data.
- 13.4% WER was achieved on a difficult YouTube video transcription task, outperforming a well-trained conventional context-dependent phone-based system.