Source-linked AI summary
The Microsoft 2016 Conversational Speech Recognition System
W. Xiong, J. Droppo, X. Huang, F. Seide, M. Seltzer, A. Stolcke, D. Yu, G. Zweig
TL;DR
The paper addresses state-of-the-art conversational speech recognition on the Switchboard benchmark. It combines diverse neural acoustic and language models with adaptation, lattice-free MMI training, rescoring, and system combination, achieving 6.9% WER for the best single system and 6.2% for the ensemble.
Problem
The paper seeks to advance conversational telephone speech recognition, a benchmark task, using improved neural acoustic and language models.
Method
The system combines CNN and LSTM acoustic-model variants with i-vector adaptation, lattice-free MMI training, recurrent language-model rescoring, and confusion-network system combination.
Results
6.2% word error rate on the Switchboard test data was achieved by the ensemble, compared with 6.9% for the best single system on the NIST 2000 Switchboard set.
Takeaways & Limitations
The ensemble advances the state of the art on the Switchboard recognition task, while CNN acoustic models and RNN language models are reported as particularly effective.
Takeaways & Limitations
Adding more hidden layers to the RNNLM produced no further gains.
Abstract
from arXiv · showhide
We describe Microsoft's conversational speech recognition system, in which we combine recent developments in neural-network-based acoustic and language modeling to advance the state of the art on the Switchboard recognition task. Inspired by machine learning ensemble techniques, the system uses a range of convolutional and recurrent neural networks. I-vector modeling and lattice-free MMI training provide significant gains for all acoustic model architectures. Language model rescoring with multiple forward and backward running RNNLMs, and word posterior-based system combination provide a 20% boost. The best single system uses a ResNet architecture acoustic model with RNNLM rescoring, and achieves a word error rate of 6.9% on the NIST 2000 Switchboard task. The combined system has an error rate of 6.2%, representing an improvement over previously reported results on this benchmark task.
1. INTRODUCTION
The system advances conversational telephone speech recognition by combining improved neural acoustic models with extensive model ensembles. Its main components include CNN and LSTM acoustic architectures and attention-based weighting of distant context.
- Convolutional and recurrent neural networks model broad acoustic context, while recurrent language models improve over classical N-gram models through continuous word representations.
- The system combines extensive model ensembles with improved component models to advance conversational telephone speech recognition.CTS has served as a benchmark speech recognition task since the 1990s.
- The acoustic-model ensemble contains CNNs and LSTMs, with multiple variants of each architecture.
- LACE uses an attention mechanism to differentially weight distant context.
4. The use of i-vector based adaptation [22] in all models
The system combines recurrent language-model rescoring with confusion-network system combination. Because many candidate systems are available, it searches for a complementary subset rather than combining all candidates.
- Multiple forward- and reverse-direction recurrent neural-network language models rescore each acoustic-model subsystem.
- Rescored N-best lists are aligned into a single confusion network for system combination.
- A greedy search selects a small, maximally complementary system subset because exhaustive combination is impractical and risks overfitting.The search starts with the single best system and adds systems successively.
- The paper describes the CNN and LSTM models, i-vector adaptation, lattice-free MMI training, language-model rescoring, experiments, related work, and conclusions.
2. CONVOLUTIONAL AND LSTM NEURAL NETWORKS
The acoustic modeling component uses several CNN architectures alongside bidirectional LSTMs. The CNN designs include VGG, ResNet, and LACE, with LACE expanding context and applying learned attention.
- Three CNN variants are used: VGG, ResNet, and LACE.The VGG model uses small 3x3 filters and deep convolutional stacks, while ResNet adds highway connections.
- LACE expands effective context layer by layer and applies a learned attention mask to weight frames differently.Its blocks use strided convolution, ReLU-convolution layers, and ResNet-like jump links.
- Bidirectional LSTMs are used without frame-skipping and provide a close second to the best-performing convolutional models.More than six layers did not improve development-set word error rate; the chosen configuration uses 512 hidden units per direction per layer.
- Figure 1 presents the LACE network architecture.
3. SPEAKER ADAPTIVE MODELING
Speaker adaptation conditions each acoustic model on a conversation-side i-vector. The implementation differs by architecture to accommodate the structure of LSTMs and CNNs.
- Each conversation side receives a 100-dimensional i-vector characterizing its speaker.
- For LSTMs, the conversation-side i-vector is appended to every input frame.
- For CNNs, a learnable layer-specific weight matrix transforms the i-vector before adding it to each layer’s pre-activation.Directly appending the vector is considered inappropriate because CNN inputs are not expected to contain spatially contiguous patterns.
4. LATTICE-FREE SEQUENCE TRAINING
The paper trains acoustic models with lattice-free sequence objectives, using unconstrained posterior computation over a finite-state search space. A mixed-history acoustic-unit language model constructs the denominator graph, simplifying training while maintaining reliable performance.
- Sequence-training objective: LFMMI training optimizes acoustic-model parameters after standard cross-entropy training.The objective is based on maximum mutual information.
- Posterior computation: Backpropagation gradients use state posteriors obtained by summing over all possible word sequences without constraints.These posteriors are computed through alpha-beta recursions over a finite-state acceptor.
- Denominator model: The mixed-history acoustic-unit language model conditions senone transitions on both senone and phone history.The authors report better performance than purely word-based or phone-based models.
- Denominator model: The denominator graph combines the mixed-history language model with HMM transition probabilities estimated from training senone sequences.This graph supports lattice-free sequence training.
- Practical implementation: The approach reduces language-model construction complexity and provides reliable training performance.The computation is performed without pruning using GPU matrix-vector operations.
5. LM RESCORING AND SYSTEM COMBINATION
The system rescorses N-best hypotheses with large N-gram and multiple forward and backward RNN language models, then combines complementary acoustic subsystems. Incremental rescoring improvements total a 9% relative error reduction over a plain RNNLM.
- Rescoring pipeline: The decoder generates lattices and 500-best lists for subsequent language-model rescoring.Initial decoding uses a pruned N-gram language model with approximately 15.9 million entries.
- RNNLM configuration: Forward and backward RNNLM log probabilities are combined during hypothesis rescoring.The models predict words in normal and reverse temporal order.
- RNNLM configuration: RNNLMs are interpolated with N-gram models and trained in two phases using out-of-domain followed by in-domain data.The second phase adapts the model toward the conversational speech domain.
- RNNLM configuration: A second non-recurrent hidden layer lowers perplexity and word error relative to a standard single-hidden-layer RNNLM.Adding more hidden layers produced no further gains.
- Performance and combination: 9% relative error reduction accumulates from the described RNNLM measures compared with a plain RNNLM.Rescoring is performed separately for each acoustic model before subsystem combination.
6. EXPERIMENTAL SETUP AND RESULTS
The evaluation uses Switchboard and Fisher conversational speech data and tests multiple acoustic architectures under standardized decoding conditions. I-vector adaptation and LFMMI training improve error rates across models, while the final comparison reports selected systems from a larger model pool.
- Experimental setup: Evaluation uses the NIST 2000 CTS test set, containing Switchboard and CallHome subsets, with Switchboard-1 from NIST 2002 for development.Training uses English CTS Switchboard and Fisher corpora.
- Experimental setup: The acoustic models use 40-dimensional log-filterbank features extracted every 10 milliseconds with a 25-millisecond analysis window.Most models use 9000 tied states, with some using 27,000 tied states.
- Acoustic-model improvements: 5–8% relative improvement comes from i-vector adaptation across systems, including CNNs.I-vector adaptation is evaluated together with LFMMI training in the acoustic-model experiments.
- Acoustic-model improvements: 7–10% further relative error reduction comes from LFMMI training consistently across acoustic-model architectures.The authors also emphasize its procedural simplicity compared with lattice writing and postprocessing.
- Comparative results: Only models receiving non-zero weight in the final combination are shown among the 15 models built.Table 5 reports word error rates for different acoustic models on the NIST 2000 CTS test set.
7. RELATION TO PRIOR WORK
The system extends prior CNN and LFMMI approaches through deeper residual-style convolutional networks, LACE attention, and a mixed triphone/phoneme-history language model. Its language-model combination produces substantial relative gains for both the best CNN and the combined system.
- Acoustic-model relation: The CNNs are deeper than earlier speech-recognition CNNs and use linear bypass connections across convolutional layers.The architectures are related to more recent CNN models in the literature.
- Acoustic-model relation: LACE expands effective context layer by layer and applies attention to weight distant context differentially.It is presented as an improvement over earlier TDNN-style architectures.
- Sequence-training relation: The LFMMI approach extends previous work by using a mixed triphone/phoneme history in the language model.This distinguishes the sequence-training formulation from earlier lattice-free approaches.
- Language-model relation: 20% relative word-error reduction comes from RNNLM rescoring for the best CNN system, while the combined system gains 16%.The rescoring combines multiple forward and backward RNNLMs.
8. CONCLUSIONS
The system achieves 6.9% error as a single system on NIST 2000 Switchboard, while acoustic-model ensembling reaches 6.2%.
- 6.9% error rate is achieved by the best single system on the NIST 2000 Switchboard set.
- 6.2% error rate is achieved on the Switchboard test data by an ensemble of acoustic models.
- CNNs in the acoustic model and RNN language models are identified as particularly effective components.