The IBM 2015 English Conversational Telephone Speech Recognition System

1. Introduction

• Switchboard became a standard LVCSR testbed for reproducible results, and techniques that improve on it tend to work on small- and large-vocabulary tasks.The authors cite LDA/STC, VTLN, FMLLR, and lattice-based discriminative training as examples.Source: §1 · p. 1
• The paper traces Switchboard WER from 43% in 1995 to 12–14% for multiple sites with deep neural networks, against an estimated human rate of around 4%.The cited historical results include 19.3% in 2000 and 15.2% for IBM in 2004.Source: §1 · p. 1
• Reaching 8.0% WER is described as nontrivial, with larger acoustic and language models trained on more data identified as a key ingredient.The authors also point to diverse neural architectures and input representations, maxout networks, language-model rescoring, and a strong GMM-HMM baseline.Source: §1 · p. 1
• The paper outlines common processing steps, system improvements, and a discussion of findings and future opportunities.The system-improvement section covers acoustic and language modeling.Source: §1 · p. 1

2.1. Training and test data

• The 1975-hour training set combines Switchboard, Fisher, and CallHome telephone conversations; Hub5 2000 evaluation contains Switchboard and CallHome test parts.The training set comprises 262 hours of Switchboard, 1698 hours of Fisher, and 15 hours of CallHome audio; the decoding vocabulary has 30.5K words.Source: §2.1 · p. 1

2.2. Feature extraction

• Features use VTL-warped PLP cepstra with LDA, semi-tied covariance, and conversation-side FMLLR; CNNs use augmented log-mel inputs, while DNNs also receive i-vectors.Each DNN block appends a 100-dimensional i-vector extracted once per conversation side.Source: §2.2 · pp. 1–2

2.3. Neural network training

• Neural networks are trained on context-dependent HMM-state targets aligned by a GMM-HMM, then undergo state-based minimum Bayes risk sequence training.The passage specifies cross-entropy training followed by 20–30 iterations of Hessian-free sequence training.Source: §2.3 · p. 2

3.1. Maxout networks with annealed dropout

• Maxout networks trained with annealed dropout showed a clear advantage over traditional sigmoid-based networks in experiments using 262 hours of SWB-1 data.The comparison covered DNNs, CNNs, and score fusion; the Maxout-AD CNN used 128 and 256 feature-map outputs, versus 512/512 for the sigmoid CNN.Source: §3.1 · p. 2, Table 1 · p. 2

3.2. Networks with very large output layers

• With 2000 hours of training data, the authors increased context-dependent HMM outputs and explored heptaphone context, then chose 32K outputs and pentaphone context for subsequent experiments.A typically 512-neuron bottleneck layer kept computation and parameter counts in check.Source: §3.2 · p. 2, Table 2 · p. 3

3.3. Improved joint training of recurrent and convolutional nets

• The joint model was initialized to match score fusion by concatenating bottleneck-to-output weight matrices and dividing by the number of models for uniform weights.The experiments used either cross-entropy-trained or sequence-trained models as initialization, then optimized lattice-based MBR loss.Source: §3.3 · p. 3
• Joint modeling and sequence-discriminative retraining improved over score fusion of sequence-trained models by 0.5% on CallHome and 0.1% on SWB.The joint-model WERs followed sequence training.Source: Table 4 · p. 4, §3.3 · p. 4

3.4. Language model

• The rebuilt 37M-n-gram 4-gram LM improved WER by 0.5% on SWB and 0.3% on CallHome; model M added improvements of 0.4% and 1.0%, respectively.The n-gram LM combined corpus-specific models with modified Kneser-Ney smoothing; 0.1–0.2% of its improvement was also attributed to a larger decoding beam.Source: §3.4 · p. 4, Table 5 · p. 4
• 0.8% on SWB (8.8% to 8.0%) and 1.2% on CallHome (15.3% to 14.1%) were gained by interpolating NNLMs with model M and the n-gram LM.The comparison is against the n-gram LM baseline.Source: Table 5 · p. 4, §3.4 · p. 4
• The authors trained NNLMs on 24M- and 560M-word datasets, using a hierarchical approximation to speed training on the larger dataset.The NNLMs were interpolated with model M and the n-gram LM.Source: §3.4 · p. 4

• The improvements substantially lowered the English Switchboard error rate relative to the previous best result.Source: §4 · p. 4
• In decreasing importance, the authors rank strong language-model rescoring, joint RNN/CNN training with 32,000 outputs on 2,000 hours of audio, then maxout nets with annealed dropout.Source: §4 · p. 4
• They expect gains from all-data maxout nets and larger 512/512-filter CNNs, and foresee human accuracy within a decade through much more data and tightly integrated knowledge sources.Source: §4 · p. 4

• Acknowledged contributors include H. Soltau, D. Povey, S. Chen, A. Emami, V. Goel, B. Kingsbury, L. Mangu, B. Ramabhadran, T. Sainath, and G. Zweig.Source: §5 · p. 4