Source-linked AI summary
Building competitive direct acoustics-to-word models for English conversational speech recognition
Kartik Audhkhasi, Brian Kingsbury, Bhuvana Ramabhadran, George Saon, Michael Picheny
TL;DR
Direct A2W models simplify speech recognition by removing decoder and language-model components, but traditionally require much more data to match sub-word systems. This paper develops a training recipe that makes A2W competitive and introduces SAR to provide readable outputs for unseen or rare words.
Problem
Prior A2W models required orders of magnitude more training data to perform comparably to conventional sub-word systems, despite simpler training and decoding.
Method
The paper develops an A2W training recipe and a spell-and-recognize CTC model that first spells words into character sequences before recognizing them.
Results
8.8%/13.9% WER on the Switchboard/CallHome Hub5-2000 subsets was achieved without a decoder or language model, while SAR provided readable output for OOV words.
Takeaways & Limitations
Model initialization, training data order, and regularization are crucial for competitive A2W performance, while SAR retains direct-A2W simplicity and improves output for unseen or rarely seen words.
Takeaways & Limitations
The direct A2W model has a fixed 25,000-word vocabulary, so its performance can be affected by OOV words on tasks with larger or different vocabularies.
Abstract
from arXiv · showhide
Direct acoustics-to-word (A2W) models in the end-to-end paradigm have received increasing attention compared to conventional sub-word based automatic speech recognition models using phones, characters, or context-dependent hidden Markov model states. This is because A2W models recognize words from speech without any decoder, pronunciation lexicon, or externally-trained language model, making training and decoding with such models simple. Prior work has shown that A2W models require orders of magnitude more training data in order to perform comparably to conventional models. Our work also showed this accuracy gap when using the English Switchboard-Fisher data set. This paper describes a recipe to train an A2W model that closes this gap and is at-par with state-of-the-art sub-word based models. We achieve a word error rate of 8.8%/13.9% on the Hub5-2000 Switchboard/CallHome test sets without any decoder or language model. We find that model initialization, training data order, and regularization have the most impact on the A2W model performance. Next, we present a joint word-character A2W model that learns to first spell the word and then recognize it. This model provides a rich output to the user instead of simple word hypotheses, making it especially useful in the case of words unseen or rarely-seen during training.
1. INTRODUCTION
Direct A2W models simplify speech recognition by mapping acoustics directly to words without sub-word units, pronunciation models, decoders, or external language models, but historically required far more data. This paper presents a training recipe that reaches competitive accuracy and a spell-and-recognize model for sparse or unseen words.
- 1. INTRODUCTION: A2W models directly optimize word probabilities from acoustics, eliminating sub-word units, pronunciation models, decision trees, decoders, and externally trained language models.This simplifies both training and decoding.
- 1. INTRODUCTION: Prior A2W systems required orders of magnitude more training data because each word needed sufficient acoustic examples.One prior system used more than 125,000 hours for a vocabulary of nearly 100,000 words.
- 1. INTRODUCTION: The spell-and-recognize CTC model first spells each word into characters and then recognizes it, retaining direct A2W decoding while producing readable hypotheses for unseen or rare words.The paper illustrates this benefit for out-of-vocabulary words.
2. BASELINE ACOUSTICS-TO-WORD MODEL
The baseline uses CTC to train a BLSTM A2W model on conversational speech, with word-level decoding by peak-picking. Its larger-data configuration still lagged behind phone CTC, motivating the updated training recipe.
- 2.1. CTC Loss: CTC removes the need for one-to-one input-output alignments by summing over valid paths containing blanks and repeated labels.A forward-backward algorithm computes the loss and gradient for backpropagation.
- 2.2. Baseline A2W Model: The experiments use 262 hours of Switchboard speech for the 300-hour set and an additional 1698 Fisher hours plus 15 CallHome hours for the 2000-hour set.The passage labels these as the “300-hour” and “2000-hour” training sets.
- 2.2. Baseline A2W Model: The baseline is a 5-layer bidirectional LSTM with 10,000-word and 25,000-word output layers for the 300-hour and 2000-hour systems, respectively.Vocabulary words were required to occur at least five times in training.
- 2.2. Baseline A2W Model: A2W decoding uses peak-picking over word posteriors followed by removal of repetitions and blanks, whereas phone CTC uses a full decoding graph and language model.This preserves the direct model’s decoder-free inference setup.
- 2.2. Baseline A2W Model: The 2000-hour A2W baseline trails phone CTC by 3.4%/2.8% absolute WER on Switchboard/CallHome, with a larger gap for the 300-hour models.These are the baseline comparisons reported on Hub5-2000.
3. UPDATED TRAINING RECIPE
The updated recipe emphasizes initialization, data ordering, regularization, and architectural choices for training A2W models, with ascending-length sorting and dropout among the evaluated strategies.
- Model initialization and regularization were identified as important aspects of training A2W models.
- 3.1. Training Data Order: Ascending-length sorting produced significantly better WER than descending-length sorting, consistent with a curriculum-learning interpretation.Shorter sequences are described as easier to train initially and potentially helpful for reaching a better parameter-space region.
- 3.2. Momentum and Dropout: Momentum and dropout both improved WER in the 300-hour A2W experiments.Dropout was used to prevent over-fitting, while momentum was implemented through a running weighted sum of loss gradients.
- A 256-dimensional projection layer sped up training by 1.2x and slightly improved WER, attributed to reduced over-fitting.
- 3.5. Final Model and Results: The recipe's 2000-hour model improved absolute WER by 4.2%/4.9% over the previous result and reached parity with most compared hybrid and end-to-end systems.The direct A2W system used no decoder or language model; the comparison is reported on Switchboard/CallHome.
- The A2W system used a 25,000-word vocabulary with 0.5%/0.8% OOV rates on the SWB/CH test sets.The paper notes that larger-vocabulary comparison systems did not incur OOV-induced errors.
4. SPELL AND RECOGNIZE MODEL
The spell and recognize (SAR) model combines direct word recognition with character spelling to address fixed-vocabulary and rare-word limitations. It preserves simple A2W decoding while producing more informative outputs for unseen or infrequent words.
- Motivation: A2W models have fixed vocabularies, cannot recognize OOV words, and recognize infrequent words poorly because training examples are insufficient.The SAR approach targets these limitations without relying on sub-word decoding pipelines.
- Model: SAR trains one model to first spell each word into characters and then recognize the word.Its target sequence interleaves character outputs with word outputs, using word-boundary prefixes and suffixes.
- Model: SAR uses a single softmax over words and characters rather than separate word and character output networks.This design avoids the synchronization problem of multitask word and character CTC outputs.
- Character set: The position-aware character set slightly outperformed the simple 41-symbol character set.It distinguishes beginning, middle, and end positions, repeated characters, and double peaks from legitimate repetitions.
- Experiments: The SAR experiments used the 300-hour dataset with a 10,000-word vocabulary and an A2W-initialized six-layer BLSTM.The setup was chosen because its smaller vocabulary produces a higher OOV rate and includes rare words.
- Decoding: SAR offers word, character, and switched decodes, with switched decoding using characters after an UNK prediction and words elsewhere.The table evaluates these three decode types for the 300-hour model.
- Results: SAR matches baseline A2W performance while providing meaningful OOV output, although some generated spellings are incorrect.For example, an OOV word may be emitted as UNK while preceding characters still provide an approximate spelling.
5. CONCLUSION
The paper shows that direct acoustics-to-word models can perform competitively with sub-word systems when trained with an appropriate recipe, while SAR adds readable output for OOV words. Its main recipe emphasizes data order, initialization, and regularization.
- Conclusion: A2W models trained on 2000-hour Switchboard+Fisher data perform at par with state-of-the-art sub-word models.The approach retains training and decoding simplicity without requiring a decoder or externally trained language model.
- Conclusion: 8.8%/13.9% WER was achieved on the Hub5-2000 Switchboard/CallHome test subsets.The reported WER corresponds to the competitive 2000-hour A2W system.
- Conclusion: Data order, model initialization, and regularization were identified as crucial factors for obtaining a competitive A2W model.These factors are presented as the key components of the training recipe.
- Conclusion: The SAR model learns to spell a word before recognizing it, producing rich and readable OOV output while retaining A2W simplicity and performance.The paper illustrates this benefit with utterances containing OOV words.