Source-linked AI summary
Achieving Human Parity in Conversational Speech Recognition
W. Xiong, J. Droppo, X. Huang, F. Seide, M. Seltzer, A. Stolcke, D. Yu, G. Zweig
TL;DR
Conversational speech recognition lacks a reliable single human benchmark because performance varies between NIST 2000’s Switchboard and CallHome subsets. The paper measures professional-transcriber error rates and develops a combined CNN/LSTM system with spatial smoothing and sequence-training methods, reaching human-parity performance on both subsets.
Problem
The paper addresses the need for reliable human error-rate benchmarks on the widely used NIST 2000 conversational speech recognition test set.
Method
The system combines CNN and LSTM acoustic models, spatial smoothing, lattice-free MMI training, recurrent language modeling, and systematic system combination.
Results
5.8% and 11.0% WER are achieved on the Switchboard and CallHome portions, slightly below professional-transcriber rates of 5.9% and 11.3%.
Takeaways & Limitations
Automatic recognition performance is reported as on par with human performance on the NIST 2000 conversational telephone speech task.
Takeaways & Limitations
The reported human measurements use standard two-pass transcription without complex multi-party transcription and adjudication.
Abstract
from arXiv · showhide
Conversational speech recognition has served as a flagship speech recognition task since the release of the Switchboard corpus in the 1990s. In this paper, we measure the human error rate on the widely used NIST 2000 test set, and find that our latest automated system has reached human parity. The error rate of professional transcribers is 5.9% for the Switchboard portion of the data, in which newly acquainted pairs of people discuss an assigned topic, and 11.3% for the CallHome portion where friends and family members have open-ended conversations. In both cases, our automated system establishes a new state of the art, and edges past the human benchmark, achieving error rates of 5.8% and 11.0%, respectively. The key to our system's performance is the use of various convolutional and LSTM acoustic model architectures, combined with a novel spatial smoothing method and lattice-free MMI acoustic training, multiple recurrent neural network language modeling approaches, and a systematic use of system combination.
1. INTRODUCTION
The paper reassesses human performance on conversational speech recognition and presents a system that reaches or slightly surpasses professional-transcriber benchmarks. Its contributions combine new human measurements, spatial regularization, language-model innovations, and system combination.
- 5.9% and 11.3% are the measured professional-transcriber error rates on the Switchboard and CallHome portions of NIST 2000, respectively.The two subsets differ substantially, making a single human error-rate figure inappropriate.
- A comprehensive assessment of human performance on the NIST eval 2000 test set establishes separate benchmarks for its conversational subsets.
- Spatial regularization significantly boosts the performance of the paper’s LSTM acoustic models.
- LSTM language models and letter-trigram input representations are used instead of RNN-LMs and conventional alternatives.
- A two-level combination of BLSTM-system variants surpasses the best previously reported results before full system combination.
- The paper analyzes human-versus-machine errors and finds substantial equivalence except for backchannel acknowledgments and hesitations.
2. HUMAN PERFORMANCE
Human performance is measured through ordinary professional transcription of the same NIST 2000 audio conditions used for automated systems. The resulting error rates differ sharply between Switchboard and CallHome, and reflect trained industry transcription rather than exhaustive adjudication.
- Two-pass transcription uses one transcriber working from scratch and a second listener correcting errors, without complex multi-party adjudication.
- The transcribers receive the same single-channel audio segments as the speech recognition system, with clearly separated speakers and non-interleaved conversation sides.
- 5.9% and 11.3% are the professional-transcriber error rates for the Switchboard and CallHome portions of NIST 2000, using the NIST scoring protocol.
- The measurements represent the error rate of trained professionals performing industry-standard transcript production.
- The CallHome and Switchboard subsets show extreme variability, so one human benchmark is inappropriate for the task.
3. CONVOLUTIONAL AND LSTM NEURAL NETWORKS
The acoustic models use several convolutional architectures alongside bidirectional LSTMs, while spatial smoothing regularizes LSTM activations. The section compares CNN designs and evaluates smoothing under a specified BLSTM configuration.
- 3.1. CNNs: Three CNN variants are used: VGG, ResNet, and LACE, combining convolutional context modeling with architecture-specific connections or attention.LACE expands context across layers using weighted transformations, a learned attention mask, and a linear pass-through.
- 3.1. CNNs: The fused ResNet–VGG model combines independently trained senone posteriors and is the best single system.
- 3.2. LSTMs: Bidirectional LSTMs process one frame at a time without frameskipping, with six or fewer layers and 512 hidden units per direction per layer.
- 3.3. Spatial Smoothing: Spatial smoothing regularizes between-layer LSTM activations by interpreting each activation vector as a two-dimensional image and penalizing high-pass-filter energy.
- 3.3. Spatial Smoothing: Table 2 evaluates smoothing on NIST 2000 CTS using a six-layer BLSTM with i-vectors, 40-dimensional filterbanks, and 512 dimensions per direction per layer.
4. SPEAKER ADAPTIVE MODELING
Speaker adaptation conditions the neural network on an i-vector representation of each conversation side, incorporating that speaker characterization differently for LSTM and convolutional models.
- A 100-dimensional i-vector is generated for each conversation side to characterize the speaker for adaptive modeling.
- For LSTMs, the conversation-side i-vector is appended to every input frame.
- For CNNs, speaker adaptation uses a learnable weight matrix because spatially contiguous speaker patterns are not expected in the input.
5. LATTICE-FREE SEQUENCE TRAINING
The paper replaces conventional lattice-based sequence training with a lattice-free MMI procedure using a mixed-history acoustic-unit language model. The resulting denominator graph is built from senone and phone histories derived from aligned training data.
- Lattice-free MMI computes training gradients from acoustic-state posteriors summed over unconstrained word sequences.The implementation uses forward-backward computation over a finite-state search representation.
- The mixed-history acoustic-unit language model conditions senone transitions on both senone and phone history.This model performed better than purely word-based or phone-based alternatives in the paper’s initial experiments.
- Forced alignment selects lexical variants and produces frame-aligned senone sequences for training.
- Consecutive occurrences of one senone are compressed before estimating the sequence language model.
- The denominator model is an unsmoothed variable-length N-gram whose history contains the previous phone and preceding senones within the current phone.
6. LM RESCORING AND SYSTEM COMBINATION
The system combines forward and backward recurrent language models with N-gram models, domain-adaptive training, and greedy subsystem combination. These additions produce substantial reductions in word error relative to an N-gram-only baseline.
- 6.1. RNN-LM setup: Forward and backward RNN language-model log probabilities are added after training models in both temporal directions.
- 6.1. RNN-LM setup: RNN and N-gram probabilities are interpolated separately by direction, with two differently initialized RNN models per direction.The interpolation weights for the two RNN models and the N-gram model are 0.375, 0.375, and 0.25.
- 6.3. Training data: RNN and LSTM language models are first trained on in-domain and out-of-domain data, then refined using in-domain data alone.In-domain validation data controls learning-rate scheduling and termination for the RNN-LM training phases.
- 6.1. RNN-LM setup: A second non-recurrent hidden layer lowers perplexity and word error relative to a standard single-hidden-layer RNN-LM.
- 6.2. LSTM-LM setup: Letter-trigram LSTM models perform slightly better than word-based LSTM models, while dropout provides no improvement.
- 6.4. RNN-LM and LSTM-LM performance: 23% relative error reduction is obtained with LSTM language models, compared with 20% using RNN language models, relative to a purely N-gram-based system.The acoustic model for this comparison was the ResNet CNN.
- 6.5. System combination: The final combination uses greedy search with hierarchically smoothed weights to limit candidate systems and reduce overfitting risk.The combined BLSTM variants differ in senones, spatial smoothing, and pronunciation dictionaries.
7. MICROSOFT COGNITIVE TOOLKIT (CNTK)
CNTK makes large-scale neural acoustic-model experimentation feasible through graph-based training and distributed 1-bit SGD. The system uses multi-GPU and multi-server computation for the 2000-hour corpus.
- CNTK scales neural-network training across multiple GPUs and servers, enabling fast experimentation with the full 2000-hour corpus.
- CNTK specifies networks and training criteria as computational formulas and trains parameters with automatic differentiation and SGD.
- 1-bit SGD distributes minibatches across worker nodes and aggregates sub-gradients to parallelize acoustic-model training.
- Automatic minibatch-size scaling reduces update frequency while selecting the largest minibatch that maintains training loss.
- Table 7 compares training, evaluation, and decoding runtimes as multiples of speech duration across acoustic-model architectures.
8. EXPERIMENTS AND RESULTS
Experiments train on English conversational speech and evaluate on the NIST 2000 CTS test set. The paper reports acoustic-model, training, and search analyses alongside comparative benchmark results.
- 8.1. Data and evaluation: The NIST 2000 CTS evaluation includes Switchboard and CallHome, while Switchboard-1 from NIST 2002 serves for tuning and development.
- 8.2. Acoustic Model Details: 5–8% relative improvement is achieved from i-vector adaptation, including on convolutional neural network systems.
- 8.3. Overall Results and Discussion: The final system combines convolutional and BLSTM acoustic models, with selected variants differing in smoothing, senone count, and pronunciation dictionary.
- 8.3. Overall Results and Discussion: BLSTM systems alone achieve an error within 3% relative of the full system combination.
- 8.3. Overall Results and Discussion: The 500-best ResNet lattices have oracle WERs of 2.7% on Switchboard and 4.9% on CallHome.
- 8.3. Overall Results and Discussion: The paper concludes that search errors are not a major limiting factor because oracle error rates are less than half the achieved actual error rates.
9. ERROR ANALYSIS
Machine and human errors are substantially similar, except for confusions involving backchannels and hesitations. Both also frequently err on short function words, while scoring conventions shape some observed substitutions.
- Backchannel–hesitation confusions are the major exception separating machine errors from human errors.The recognizer often substitutes a backchannel for a hesitation, whereas people generally do not show this pattern.
- Hesitations and backchannels serve opposite turn-management functions in conversation.Backchannels acknowledge the other speaker and signal continuation, while hesitations indicate that the current speaker wants to retain the turn.
- NIST scoring treats hesitation words as optional, so recognizer hesitations are removed before scoring.This produces many substitutions of backchannels for hesitations but not the reverse pattern.
- Short function words dominate insertion and deletion errors for both people and the automated system.The most common transcriber error was omitting the word “I.”
10. RELATION TO PRIOR WORK
The paper builds on deeper CNN speech-recognition architectures by adding LACE's expanding context and attention weighting, alongside a spatial regularization technique.
- The CNNs are deeper than earlier speech-recognition CNNs and use linear bypass connections across convolutional layers.These architectures are related to more recent CNN designs cited by the paper.
- LACE expands the effective context window layer by layer and uses an attention mask to weight distant context differently.
- The paper's spatial regularization technique is presented as an improvement to these convolutional architectures.
11. CONCLUSIONS
The paper measures human error on NIST 2000 conversational telephone speech and reports automatic recognition performance on par with human performance. Its performance is attributed to combined CNN and LSTM acoustic and language modeling systems.
- 5.8% and 11.0% error rates are reported for the Switchboard and CallHome subsets, respectively.The paper also reports substantial variability between the two subsets.
- Automatic recognition performance is reported as on par with human performance on the NIST 2000 conversational telephone speech task.
- System performance is attributed to LSTMs for acoustic and language modeling, CNNs in the acoustic model, and extensive combination of complementary systems.