Source-linked AI summary
The Microsoft 2017 Conversational Speech Recognition System
W. Xiong, L. Wu, F. Alleva, J. Droppo, X. Huang, A. Stolcke
TL;DR
The paper addresses continued improvement of conversational speech recognition on the Switchboard task. It updates acoustic and language modeling, system combination, and confusion-network rescoring, achieving a 5.1% overall error rate and surpassing the previously measured human accuracy level.
Problem
The paper seeks to advance conversational speech recognition beyond the previously measured 5.9% human benchmark.
Method
The system adds CNN-BLSTM acoustics, character-based and session-level LSTM language models, two-stage acoustic-model combination, and post-combination confusion-network rescoring.
Results
5.1% overall error rate was achieved, surpassing the previously measured human accuracy level.
Takeaways & Limitations
The updated system reduced error rate for the Switchboard tasks by 12% relative.
Takeaways & Limitations
Further evaluation, including on the CallHome speech genre, remains future work.
Abstract
from arXiv · showhide
We describe the 2017 version of Microsoft's conversational speech recognition system, in which we update our 2016 system with recent developments in neural-network-based acoustic and language modeling to further advance the state of the art on the Switchboard speech recognition task. The system adds a CNN-BLSTM acoustic model to the set of model architectures we combined previously, and includes character-based and dialog session aware LSTM language models in rescoring. For system combination we adopt a two-stage approach, whereby subsets of acoustic models are first combined at the senone/frame level, followed by a word-level voting via confusion networks. We also added a confusion network rescoring step after system combination. The resulting system yields a 5.1\% word error rate on the 2000 Switchboard evaluation set.
1. INTRODUCTION
Microsoft’s 2017 conversational speech recognition system extends its earlier system with additional neural acoustic and language models, refined system combination, and post-combination rescoring. These changes produced a 5.1% overall error rate, surpassing the previously measured human accuracy level.
- Conversational speech recognition had advanced through deep networks, efficient training, and neural acoustic and language modeling.
- 5.9% human word error was not statistically different from the earlier system’s 5.8% error rate on the Switchboard evaluation data.Human and machine transcriptions also showed remarkably similar qualitative error patterns.
- The updated system added a CNN-BLSTM acoustic model, character-based and dialog session-based LSTM language models, and refined acoustic-model combination.Acoustic predictions were combined at senone/frame and word levels, with confusion-network rescoring added afterward.
- 5.1% overall error rate surpassed the human accuracy level measured previously.
2. ACOUSTIC MODELS
The acoustic-modeling system combines CNN, BLSTM, and newly added CNN-BLSTM architectures, while increasing diversity through varied senone sets and combining models at the frame level.
- 2.1. Convolutional neural nets: The system uses ResNet and LACE CNN architectures, with batch normalization before ReLU activations in ResNet.LACE expands context through attention-weighted nonlinear transformations and ResNet-like linear pass-through connections.
- 2.2. Bidirectional LSTM: BLSTM acoustic models use bidirectional LSTM networks without frame-skipping, with six or fewer layers and 512 hidden units per direction per layer.The selected configuration balances training time and final model accuracy.
- 2.3. CNN-BLSTM: The added CNN-BLSTM applies three convolutional layers to context-window features before six BLSTM layers process the resulting time sequence.Each input feature uses a [−3, 3] context window and has size 40x7xt, with frequency-dimension zero-padding.
- 2.4. Senone set diversity: Acoustic-model diversity comes from four senone sets differing in clustering detail, phone sets, and dictionaries.The sets include 9k versus 27k senones and alternate phones for filled pauses and backchannel words.
- 2.6. Sequence training: The models are sequence-trained with MMI, using a full phone-and-senone trigram denominator graph whose computations run efficiently on GPUs.The forward-backward computations are expressed as matrix operations without lattice approximation.
- 2.7. Frame-level model combination: Frame-level combination averages senone posteriors from multiple acoustic models, requiring identical underlying senone sets.The subsequent evaluations compare senone sets, architectures, and equal-weighted frame-level combinations using N-gram language models.
3. LANGUAGE MODELS
The updated language-modeling pipeline expands vocabulary, adds character- and session-based LSTM-LMs, and combines them through log-linear rescoring. Session modeling captures global and local conversational context and substantially reduces perplexity, though noisy recognition history reduces the gain.
- Vocabulary size: Vocabulary expansion from 30,500 to 165k words reduced the out-of-vocabulary rate by about 0.2% absolute.The added words came from frequent terms in the Broadcast News and UW Conversational Web corpora.
- LSTM-LM rescoring: The updated system adds character-based, letter-trigram word-based, and subset-trained letter-trigram word-based LSTM-LMs.These variants supplement the existing utterance-scoped LSTM-LM models.
- LSTM-LM rescoring: Forward and backward LSTM-LMs are combined additively in log-probability space, while different architectures are combined log-linearly during rescoring.The log-linear optimization jointly tunes language-model and other knowledge-source weights.
- Dialog session-based modeling: Session-based LSTM-LMs serialize both speakers' utterances by onset time to predict each following word using global and local conversation history.Optional inputs encode speaker changes and temporal overlap; 1-best hypotheses provide history during speech-test evaluation.
- Dialog session-based modeling: A 21% perplexity reduction comes from conditioning on previous-word context, while noisy 1-best history leaves only a 7-8% relative perplexity degradation.Adding speaker-change and overlap information provides smaller incremental reductions.
4. EXPERIMENTAL SETUP
The experiments retain established conversational-speech datasets and evaluate on the NIST 2000 Switchboard test set. Neural networks are trained with CNTK on distributed multi-GPU infrastructure to make large-corpus training feasible.
- Data and evaluation: Acoustic training uses the English Switchboard and Fisher corpora, while language modeling also uses CallHome, Switchboard-2, Broadcast News, and UW conversational web transcripts.Evaluation is performed on the Switchboard portion of the NIST 2000 CTS test set.
- Training infrastructure: CNTK training runs on a Linux multi-GPU server farm and scales across multiple GPUs and servers.The resulting experimental turnaround was important for training with the full 2000-hour corpus.
- Training infrastructure: A 1-bit SGD data-parallel method distributes minibatches across workers and aggregates subgradients to avoid prohibitive communication costs.Without this parallelization, training the acoustic models on one GPU would take many weeks or months.
- Optimization: Individual acoustic-model training converges after 1.5 to 2 passes over the 2000-hour training set using FsAdaGrad with automatically adjusted learning rates.The reported typical learning rate is 3 × 10^-6, with a decrease factor of 0.7.
5. SYSTEM COMBINATION AND RESULTS
The system combines rescored outputs in confusion networks after log-linear score integration, then applies a final confusion-network rescoring stage. Language models, system combination, and final rescoring each contribute measured error reductions.
- System combination: All rescored system outputs are combined log-linearly, normalized into utterance-level posteriors, and merged into a single word confusion network.The merged N-best outputs are processed with the SRILM nbestrover tool.
- System combination: A brute-force development-set search selects seven systems from nine candidates, with equal system weights during combination.Two systems are eliminated from the initial set of nine.
- Confusion-network rescoring: Final confusion-network rescoring regenerates N-best lists and combines N-gram, utterance-level LSTM, and an additional knowledge source using confusion-network word log posteriors as acoustic scores.This stage follows the initial system combination.
- Confusion-network rescoring: A learned penalty for backchannel counts targets the machine-specific confusion of filled pauses such as ‘uh’ and ‘um’ with backchannel acknowledgments.The optimized weight for backchannel count is negative.
- Results: LSTM-LM rescoring reduces individual-system error by 22 to 25% relative, system combination adds 4%, and confusion-network rescoring adds another 2-3% relative.The progression is measured from N-gram-rescored individual systems through combination and final rescoring.
6. CONCLUSIONS AND FUTURE WORK
The system combines enhanced acoustic and language modeling with two-stage system combination and confusion-network rescoring. It reduces Switchboard error from 5.8% to 5.1% relative to the 2016 system, while broader evaluation remains future work.
- Acoustic models are combined first at the senone/frame level, then through word-level confusion-network combination.
- 12% relative error reduction lowers Switchboard error from 5.8% for the 2016 system to 5.1%.
- The language model uses larger vocabulary, additional LSTM variants, and session-level modeling of coherence and dialog phenomena.The session-level model gives over 20% relative perplexity reduction.
- A final confusion-network rescoring step with special treatment for backchannels provides a small additional gain after system combination.
- More thorough evaluation, including the CallHome speech genre, is planned as future work.