Source-linked AI summary
Deep Speech: Scaling up end-to-end speech recognition
Awni Hannun, Carl Case, Jared Casper, Bryan Catanzaro, Greg Diamos, Erich Elsen, Ryan Prenger, Sanjeev Satheesh, Shubho Sengupta, Adam Coates, Andrew Y. Ng
TL;DR
Speech recognition traditionally depends on complex engineered pipelines that are difficult to make robust to noise and speaker variation. Deep Speech uses end-to-end RNN learning with multi-GPU training and collected or synthesized distorted speech, achieving strong results on clear and noisy recognition tasks, including 16.0% error on Switchboard Hub5’00.
Problem
Traditional speech systems rely on heavily engineered pipelines, while improving robustness to noisy speech requires further laborious engineering.
Method
Deep Speech trains an end-to-end RNN using multi-GPU computation and collected or synthesized speech data containing noise and speaker variation.
Results
16.0% error was achieved on the full Switchboard Hub5’00 test set, and Deep Speech outperformed commercial systems in noisy speech recognition tests.
Takeaways & Limitations
The paper presents a speech system that outperforms existing state-of-the-art pipelines in clear conversational speech and noisy environments without their complex processing stages.
Takeaways & Limitations
Synthesizing noisy speech requires many distinct noise clips because repeating a short noise track could let the recurrent network memorize and subtract it.
Abstract
from arXiv · showhide
We present a state-of-the-art speech recognition system developed using end-to-end deep learning. Our architecture is significantly simpler than traditional speech systems, which rely on laboriously engineered processing pipelines; these traditional systems also tend to perform poorly when used in noisy environments. In contrast, our system does not need hand-designed components to model background noise, reverberation, or speaker variation, but instead directly learns a function that is robust to such effects. We do not need a phoneme dictionary, nor even the concept of a "phoneme." Key to our approach is a well-optimized RNN training system that uses multiple GPUs, as well as a set of novel data synthesis techniques that allow us to efficiently obtain a large amount of varied data for training. Our system, called Deep Speech, outperforms previously published results on the widely studied Switchboard Hub5'00, achieving 16.0% error on the full test set. Deep Speech also handles challenging noisy environments better than widely used, state-of-the-art commercial speech systems.
1 Introduction
Deep Speech replaces hand-engineered speech-recognition pipelines with an end-to-end RNN trained using large-scale data and multi-GPU computation. The approach targets robustness to noise and speaker variation while achieving strong performance on difficult speech tasks.
- Deep Speech applies end-to-end deep learning to produce speech transcriptions without relying on traditional hand-engineered processing stages.The system learns directly from data rather than using specialized components for speaker adaptation or noise filtering.
- The authors address these challenges with a GPU-oriented RNN, multi-GPU training, and collected and synthesized speech data containing relevant distortions.The paper describes these as the key ideas developed in its training, optimization, and data-strategy sections.
- End-to-end training can learn robustness to noise and speaker variation when sufficient data and computing power are available.
- Building large end-to-end speech systems requires large labeled datasets and networks capable of using them effectively.The paper identifies transcript alignment, training scale, and computation as central challenges.
- 16.0% error on the full Switchboard Hub5’00 test set was reported as the best published result, alongside better performance than commercial systems in noisy tests.
2 RNN Training Setup
Deep Speech uses a compact RNN to transform speech spectrograms into character probabilities, then combines them with a language model for word-level decoding. The design emphasizes simple recurrent computation, CTC training, and variance-reduction techniques.
- RNN architecture: The RNN ingests spectrogram time series and predicts character probabilities for each time slice.Each character distribution includes letters, space, apostrophe, and the CTC blank symbol.
- RNN architecture: The model has five hidden layers: three feedforward layers, one bidirectional recurrent layer, and a fifth layer combining forward and backward activations.The first layer uses neighboring spectrogram frames as temporal context, while later feedforward layers operate independently at each time step.
- Training objective: The output layer applies a softmax over characters, and CTC loss supplies gradients for back-propagation through the network.Training uses Nesterov’s Accelerated gradient method after computing the loss from predicted and ground-truth character sequences.
- Architectural choices: The homogeneous single-recurrent-layer design avoids LSTM gating overhead and makes computation simpler, though recurrent forward and backward passes remain sequential.The authors state that this structure is considerably simpler than related models and helps avoid a computational bottleneck.
- Regularization and evaluation: Dropout, audio time-shifting, and output averaging are used to reduce variance and improve evaluation robustness.Dropout is applied in feedforward layers, while test-time audio is shifted by 5ms in both directions and the resulting probabilities are averaged.
- Language-model decoding: A language-model beam search combines RNN probabilities with an N-gram model and sentence-length control to select the most probable character sequence.The weighting parameters are chosen by cross-validation, and typical beam sizes range from 1000 to 8000.
3 Optimizations
The system is optimized for large-scale multi-GPU training despite sequential recurrent computations. Parallelism, length-based batching, workload partitioning, striding, and efficient libraries reduce the associated computational costs.
- Computational design: Nearly 5 billion connections in a typical unrolled utterance make efficient computation critical for feasible training.The model uses homogeneous rectified-linear networks and a small number of optimized BLAS calls.
- Data parallelism: Multi-GPU training accelerates experiments through data parallelism, with each GPU processing a separate minibatch before gradients are combined.The authors typically use 2× or 4× data parallelism across GPUs.
- Data parallelism: Sorting utterances by length and padding with silence makes differently sized examples compatible with batched matrix multiplication.Only similarly sized utterances are combined into minibatches to address variable sequence lengths.
- Data parallelism: Training speedups show diminishing returns because larger batches do not proportionally improve convergence and smaller per-GPU batches reduce efficiency.Processing twice as many examples on twice as many GPUs does not yield a twofold training speedup.
- Recurrent-layer parallelization: The recurrent layer is partitioned across GPUs by splitting the time dimension and exchanging intermediate forward and backward activations at the midpoint.This avoids the substantial transfers caused by naively placing the two recurrent directions on separate GPUs.
- Striding: Stride-2 processing shortens the recurrent sequence by half, while cuDNN efficiently implements the first convolutional layer.The stride is applied to the original input and is analogous to a convolutional network with step size 2.
4 Training Data
Deep Speech assembles large labeled speech resources with synthetic distortions to train robustness to noisy conditions and speaker variation. The training data combines collected recordings, superimposed noise, frequency-matched clips, and intentionally induced Lombard speech.
- 5000 hours of read speech from 9600 speakers provide the extensive labeled dataset used to train the largest models.
- Synthetic data expands training coverage because labeled recordings from noisy environments are impractical to capture at scale.
- Noisy speech is synthesized by superimposing speech and noise tracks, with optional reverberation, echoes, or damping applied to their power spectra.
- Repeated noise can be memorized, so the system uses many shorter noise clips instead of repeating a single 1000-hour track.
- Candidate noise clips are filtered by frequency-band average power to better match noise recorded in real environments.
- The Lombard Effect is induced by playing loud background noise through headphones while speakers record utterances, capturing noise-related pitch and inflection changes.
5 Experiments
Experiments evaluate Deep Speech on full Hub5’00 conversational speech and a constructed noisy-speech benchmark. The system improves on a published Switchboard baseline and synthetic-noise training substantially reduces noisy-speech error.
- Conversational speech: Switchboard Hub5’00 (full): The full Hub5’00 test set is used because it combines the easier Switchboard and harder CallHome subsets.
- Conversational speech: Switchboard Hub5’00 (full): Deep Speech compares models trained on 300 hours of Switchboard with models trained on the combined 2300 hours of Switchboard and Fisher data.
- Conversational speech: Switchboard Hub5’00 (full): 2.4% absolute WER and 13.0% relative improvement over the best previously published baseline result from training on the combined 2300-hour corpus.
- Noisy speech: Noisy-speech API comparisons score only utterances for which all systems return non-empty results, excluding cases where an API fails to respond.
- Noisy speech: The noisy-trained model reaches 22.6% WER on noisy utterances versus 28.7% for the clean-trained model, a 6.1% absolute and 21.3% relative improvement.
6 Related Work
Earlier speech systems incorporated neural networks mainly as components within engineered pipelines, whereas this work emphasizes end-to-end processing. The approach also depends on scalable computation, large labeled datasets, and data synthesis.
- Earlier neural and connectionist systems replaced only one stage of the speech recognition pipeline.
- GPU-oriented designs and parallel processors are established approaches for improving large-scale deep-learning efficiency.
- Speech recognition has fewer large labeled datasets than computer vision, motivating reliance on synthesis alongside collected utterances.
7 Conclusion
Deep Speech outperforms existing state-of-the-art recognition pipelines in clear conversational speech and noisy environments. Multi-GPU training and collected and synthesized distortion-rich data enable this simpler, data-driven system.
- Deep Speech outperforms existing state-of-the-art recognition pipelines in clear conversational speech and noisy environments.
- Multi-GPU training and data collection and synthesis strategies enable large training sets containing background noise and Lombard effect distortions.
- The resulting data-driven system performs better without relying on the complex processing stages used by existing methods.
- The authors expect further improvement as computing power and dataset sizes increase.