Source-linked AI summary
English Conversational Telephone Speech Recognition by Humans and Machines
George Saon, Gakuto Kurata, Tom Sercu, Kartik Audhkhasi, Samuel Thomas, Dimitrios Dimitriadis, Xiaodong Cui, Bhuvana Ramabhadran, Michael Picheny, Lynn-Li Lim, Bergul Roomi, Phil Hall
TL;DR
The paper asks how human conversational-speech recognition performance should be measured and whether ASR has reached human parity. It independently measures human transcription and improves acoustic and language models, obtaining 5.5%/10.3% WER on SWB/CH while concluding that human parity has not been reached overall.
Problem
Human performance on conversational speech was uncertain because a prior study claimed human parity while reporting higher WER than another estimate.
Method
The paper independently measures human transcription on Switchboard and CallHome and combines improved LSTM, speaker-adversarial, ResNet, recurrent, and convolutional language models.
Results
5.5%/10.3% WER was achieved on the Switchboard/CallHome subsets, while the best human results were 5.1% SWB and 6.8% CallHome WER after quality checks.
Takeaways & Limitations
Human parity is attainable on this Switchboard subset but remains distant on CallHome, and the paper concludes that human parity has not been reached overall.
Takeaways & Limitations
The SWB testset has substantial training-test speaker overlap, making its ASR performance deceptively good; average WER across testsets is around 8%.
Abstract
from arXiv · showhide
One of the most difficult speech recognition tasks is accurate recognition of human to human communication. Advances in deep learning over the last few years have produced major speech recognition improvements on the representative Switchboard conversational corpus. Word error rates that just a few years ago were 14% have dropped to 8.0%, then 6.6% and most recently 5.8%, and are now believed to be within striking range of human performance. This then raises two issues - what IS human performance, and how far down can we still drive speech recognition error rates? A recent paper by Microsoft suggests that we have already achieved human performance. In trying to verify this statement, we performed an independent set of human performance measurements on two conversational tasks and found that human performance may be considerably better than what was earlier reported, giving the community a significantly harder goal to achieve. We also report on our own efforts in this area, presenting a set of acoustic and language modeling techniques that lowered the word error rate of our own English conversational telephone LVCSR system to the level of 5.5%/10.3% on the Switchboard/CallHome subsets of the Hub5 2000 evaluation, which - at least at the writing of this paper - is a new performance milestone (albeit not at what we measure to be human performance!). On the acoustic side, we use a score fusion of three models: one LSTM with multiple feature inputs, a second LSTM trained with speaker-adversarial multi-task learning and a third residual net (ResNet) with 25 convolutional layers and time-dilated convolutions. On the language modeling side, we use word and character LSTMs and convolutional WaveNet-style language models.
1. Introduction
The paper remeasures human performance on conversational telephone speech and finds lower CallHome error than previously reported. It also presents acoustic and language-model improvements that approach human performance on Switchboard but not CallHome.
- 5.1% SWB WER was the best transcriber’s result after quality checks, versus an earlier 4% estimate from another study.
- 6.8% CallHome WER was achieved by the same transcriber, substantially below the previously reported 11.3% estimate.
- 5.5%/10.3% WER on SWB/CH was achieved by the latest ASR system, making human parity attainable on SWB but distant on CallHome.
- SWB evaluation is confounded by training-test speaker overlap, while CallHome is harder because its casual speech mismatches predominantly Switchboard-like training data.
- Acoustic gains came from improved LSTMs, speaker-adversarial multi-task learning, and ResNets, while language gains came from word and character LSTMs and WaveNet-style models.
2. Human transcription experiments
The human transcription study used independent transcription and quality checking to measure error rates under a controlled protocol. Error rates varied across transcribers, quality checking reduced them, and CallHome differences from prior work were linked to deletion rates.
- Three independent transcribers produced transcripts that were quality checked by a fourth senior transcriber, with all transcribers being native US English speakers.
- 12-14xRT was estimated for the first transcription pass, followed by 1.7-2xRT for quality checking, with repeated listening in both passes.
- Non-speech markers, punctuation, partial words, and other specified markers were removed or treated as non-lexical before scoring.
- Quality checking reduced the error rate across all transcribers, although performance varied among them.
- 6.8% versus 11.3% CallHome human WER was associated with a much lower deletion rate in the best quality-checked transcript.
3. System improvements
The system improvements combine recurrent and convolutional acoustic models with adversarial training, feature fusion, ResNets, and language-model rescoring across multiple conversational testsets.
- Acoustic modeling: The system replaces maxout RNNs and VGG networks with improved LSTMs and ResNets while retaining recurrent–convolutional model combination.
- LSTM acoustic models: Speaker-adversarial multi-task learning produces small gains across all testsets, with additional gains from LSTM-based VTLN and FMLLR reestimation.
- LSTM acoustic models: Feature fusion yields the largest LSTM-modeling improvement by combining FMLLR, i-vector, and Logmel+∆+∆∆ inputs, without requiring CTC modeling.
- ResNet acoustic models: The selected ResNet uses residual blocks and time-aware sequence convolutions, while Table 6 compares architectures, training regimes, and test-set results.
- Model combination and language modeling: LSTM–ResNet frame-level score fusion improves WER across all testsets, and combined language models reach 5.5%/10.3% WER on SWB/CH.
4. Conclusion
The paper combines recurrent, convolutional, and language-modeling improvements to achieve a new Switchboard WER record, while concluding that human parity has not yet been reached.
- New acoustic and language-modeling improvements produced a record word error rate on the Switchboard task.The system combines bidirectional LSTMs, residual nets, word and character LSTM language models, and a convolutional WaveNet language model.
- Feature fusion and replacing VGG nets with residual nets were instrumental acoustic improvements with complementary accuracy benefits across testsets.
- Word and character-based LSTM language models plus a convolutional WaveNet language model exploited complementarity between recurrent and convolutional architectures.
- Human parity has not been reached because speaker overlap makes Switchboard performance deceptively strong and expert-transcriber WER is lower than previously reported.The paper reports an average system WER of around 8% across all testsets as a more realistic performance level.