PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep Speech 2: End-to-End Speech Recognition in English and Mandarin

Dario Amodei, Rishita Anubhai, Eric Battenberg, Carl Case, Jared Casper, Bryan Catanzaro, Jingdong Chen, Mike Chrzanowski, Adam Coates, Greg Diamos, Erich Elsen, Jesse Engel, Linxi Fan, Christopher Fougner, Tony Han, Awni Hannun, Billy Jun, Patrick LeGresley, Libby Lin, Sharan Narang, Andrew Ng, Sherjil Ozair, Ryan Prenger, Jonathan Raiman, Sanjeev Satheesh, David Seetapun, Shubho Sengupta, Yi Wang, Zhiqian Wang, Chong Wang, Bo Xiao, Dani Yogatama, Jun Zhan, Zhenyao Zhu

arXiv:1512.02595v1cs.CL

TL;DR

Speech recognition pipelines require many components to handle variation across languages, speakers, environments, and noise. Deep Speech 2 uses end-to-end deep learning to recognize English and Mandarin, approaching or exceeding human-worker transcription accuracy on several benchmarks while supporting production deployment.

  • Problem

    ASR systems rely on complex, application-specific components that generalize poorly across languages and acoustic conditions, motivating a more general single-engine approach.

  • Method

    Deep Speech 2 replaces most ASR pipeline modules with one end-to-end neural model and investigates architecture, data, and computational scaling for English and Mandarin.

  • Results

    Deep Speech 2 approaches or exceeds Amazon Mechanical Turk human-worker accuracy on several English and Mandarin benchmarks and is deployable in production.

  • Takeaways & Limitations

    The results support a generic speech-recognition approach that can be applied across two very different languages with little language-specific expertise and efficient GPU deployment.

  • Takeaways & Limitations

    The reported latency excludes internet transmission and the time from utterance onset to the first transcription.

Abstract

from arXiv · show

We show that an end-to-end deep learning approach can be used to recognize either English or Mandarin Chinese speech--two vastly different languages. Because it replaces entire pipelines of hand-engineered components with neural networks, end-to-end learning allows us to handle a diverse variety of speech including noisy environments, accents and different languages. Key to our approach is our application of HPC techniques, resulting in a 7x speedup over our previous system. Because of this efficiency, experiments that previously took weeks now run in days. This enables us to iterate more quickly to identify superior architectures and algorithms. As a result, in several cases, our system is competitive with the transcription of human workers when benchmarked on standard datasets. Finally, using a technique called Batch Dispatch with GPUs in the data center, we show that our system can be inexpensively deployed in an online setting, delivering low latency when serving users at scale.

1 Introduction

Deep Speech 2 demonstrates that end-to-end deep learning can replace much of the hand-engineered ASR pipeline while addressing speech variability across environments, accents, noise, and languages. The system’s advances combine model architecture, large labeled datasets, computational scale, and efficient deployment.

  • Motivation: End-to-end deep learning replaces most hand-engineered ASR modules with a single model, simplifying development across environments and languages.Traditional systems require complex feature extraction, acoustic, language, pronunciation, and adaptation components that are difficult to build and tune.
  • Motivation: Deep Speech 2 targets robustness to variation in environment, speaker accent, and noise with a single speech-recognition engine.The introduction frames human transcription robustness as the expectation for speech-recognition users.
  • Contributions: Deep Speech 2 focuses performance gains on model architecture, large labeled training datasets, and computational scale.The paper presents contributions across all three areas, including extensive architecture investigation.
  • Training data: 11,940 hours of English speech and 9,400 hours of Mandarin speech train the systems, with data synthesis used for further augmentation.These datasets are described as larger than those typically used for speech-recognition training.
  • Deployment: 67 milliseconds is the 98th-percentile compute latency achieved by the Mandarin engine with 10 simultaneous audio streams.Batch Dispatch enables an efficient, real-time implementation on production servers.

2 Related Work

Deep Speech 2 builds on established deep-learning and speech-recognition methods, including neural acoustic models, end-to-end sequence mapping, CTC-RNNs, and scalable GPU training. It also draws on prior work using large datasets, data augmentation, and speech systems to bootstrap additional labeled data.

  • Deep learning and speech recognition: Deep neural networks became standard components of state-of-the-art automatic speech recognition, following earlier use of feed-forward, recurrent, and convolutional networks.Feed-forward acoustic models, recurrent networks, and convolutional networks had all been explored for speech recognition more than 20 years ago.
  • End-to-end speech recognition: End-to-end speech recognition maps variable-length audio directly to variable-length transcriptions using RNN encoder-decoder models or CTC coupled with recurrent networks.Encoder-decoder systems use an encoder to create a fixed-length vector and a decoder to produce output predictions; CTC-RNNs model temporal information.
  • End-to-end speech recognition: CTC-RNN systems perform well with grapheme outputs and can predict phonemes, although phoneme systems still require a lexicon and have required DNN pre-training.The passage identifies CTC-RNN as a commonly used end-to-end technique for variable-length speech-to-text mapping.
  • Scaling deep learning: Deep-learning speech recognition has benefited from scale through efficient GPU primitives and model-parallel, data-parallel, or combined training approaches.Prior work scaled training from one GPU to two or more GPUs, while improving individual GPU efficiency for low-level operations.
  • Data and augmentation: Large labeled datasets, data augmentation, and system-assisted bootstrapping have been used to improve end-to-end speech recognition by increasing effective labeled data.Prior systems used over 7000 hours of labeled speech, augmentation from noise addition to vocal-tract perturbations, and recognizers to align, filter, or transcribe new speech.

3 Model Architecture · 3.1 Preliminaries

DS2 increases model capacity through deep convolutional and recurrent architectures, using spectrogram inputs and character-level outputs for English and Mandarin. The system is trained with CTC and evaluated with language-specific error-rate metrics.

  • 3 Model Architecture: Up to 11 layers increase capacity for large speech datasets, with bidirectional recurrent and convolutional layers providing nearly 8× the computation per example of DS1 models.The increased computational cost makes fast optimization and computation critical.
  • 3.1 Preliminaries: Power-normalized audio spectrograms represent each utterance as a time series of feature vectors over successive frames.Each feature vector contains the power of frequency bins in an audio frame.
  • 3.1 Preliminaries: The network predicts graphemes at each output step, using English characters plus space, apostrophe, and blank symbols or simplified Chinese characters for Mandarin.The blank symbol supports the character-level output formulation.
  • 3.1 Preliminaries: The DS2 architecture uses convolutional layers followed by recurrent layers and fully connected layers to transform speech spectrograms into text transcriptions.The core system is an RNN that ingests speech spectrograms and generates transcriptions.
  • 3.1 Preliminaries: Convolutional layers apply context-window filters with clipped ReLU nonlinearities and may stride across frames to shorten the recurrent layers’ time sequences.The clipped ReLU is σ(x) = min{max{x, 0}, 20}.
  • 3.1 Preliminaries: The recurrent stack uses forward and backward activations, supports standard recurrence, LSTM, or GRU operations, and combines the two directions by summation.The bidirectional layers follow the convolutional layers.
  • 3.1 Preliminaries: Architecture variants vary convolutional depth from 1 to 3 layers and recurrent or GRU depth from 1 to 7 layers.These variants are explored for training on both English and Mandarin speech.
  • 3.1 Preliminaries: The model applies fully connected layers and a character-level softmax output, then trains with CTC loss and backpropagation through time.The reported metrics are WER for English and CER for Mandarin, using an English held-out development set of 2048 utterances.

3.2 Batch Normalization for Deep RNNs · 3.3 SortaGrad

Deep Speech 2 uses sequence-wise BatchNorm to improve optimization and final generalization in very deep simple RNNs, while SortaGrad orders early training examples by utterance length to address CTC’s length-dependent difficulty. Together, these methods accelerate or stabilize training, though BatchNorm can hurt shallow-network generalization and requires special deployment handling.

  • 3.2 Batch Normalization for Deep RNNs: Deep networks are scaled by adding hidden layers, and BatchNorm is explored to address optimization issues in deep bidirectional RNNs.The approach increases depth rather than layer width to scale with the training set.
  • 3.2 Batch Normalization for Deep RNNs: BatchNorm substantially improves final generalization error while greatly accelerating training for very deep simple RNNs on large datasets.This contrasts with prior recurrent-network results showing faster convergence without improved generalization.
  • 3.2 Batch Normalization for Deep RNNs: Single-time-step and cumulative-time-step normalization do not improve optimization, with the latter also greatly complicating backpropagation.Sequential dependence prevents averaging statistics across all time-steps in the single-time-step approach.
  • 3.2 Batch Normalization for Deep RNNs: Sequence-wise normalization computes statistics across minibatch items and sequence lengths, making deep networks converge faster.Its performance advantage grows with network depth, reaching a 12% difference for the deepest network.
  • 3.2 Batch Normalization for Deep RNNs: BatchNorm harms generalization error in the shallowest network, which also converges more slowly than deeper networks.The comparison controls for model size by holding the total number of parameters constant.
  • 3.2 Batch Normalization for Deep RNNs: For deployment on single utterances, training-time sequence normalization is replaced by running training averages of each neuron’s mean and variance.Normalizing each neuron using only the evaluation sequence degrades performance.
  • 3.3 SortaGrad: SortaGrad uses utterance length as a difficulty heuristic because CTC costs increase with sequence length as alignment probabilities are multiplied across time.Long utterances therefore have higher cost than short utterances.
  • 3.3 SortaGrad: During the first epoch, SortaGrad presents minibatches in increasing order of their longest utterance, then returns to random minibatch order.Its training-cost effect is particularly pronounced without BatchNorm because those networks are numerically less stable.

3.4 Comparison of simple RNNs and GRUs · 3.5 Frequency Convolutions · 3.6 Striding

The paper compares GRUs with simple RNNs, explores frequency convolutions, and uses striding to trade recognition accuracy for computational efficiency. GRUs perform better at fixed parameter counts, while convolutional and bigram-striding designs improve robustness or reduce sequence length under specific conditions.

  • 3.4 Comparison of simple RNNs and GRUs: The modified GRU computes the transformed previous hidden state before applying the reset gate, allowing all hidden-state operations to use one matrix multiplication.The output nonlinearity is typically tanh, although clipped-ReLU has similar performance.
  • 3.4 Comparison of simple RNNs and GRUs: For a fixed number of parameters, GRUs achieve better WER than simple RNNs at every network depth.Both architectures benefit from batch normalization and perform strongly in deep networks.
  • 3.4 Comparison of simple RNNs and GRUs: GRUs with 5 or more recurrent layers do not significantly improve performance, likely because hidden layers are thinned to keep parameter counts constant.The hidden size decreases from 1728 units with 1 recurrent layer to 768 units with 7 recurrent layers.
  • 3.4 Comparison of simple RNNs and GRUs: At larger model sizes and a fixed computational budget, simple RNNs perform slightly better, so most later experiments use simple RNN layers.This reverses the comparison observed in Table 3, where GRUs outperform simple RNNs.
  • 3.5 Frequency Convolutions: Frequency convolutions are most effective near the input because later fully connected and recurrent layers remove spectral ordering.They model speaker-related spectral variance more concisely than large fully connected networks.
  • 3.5 Frequency Convolutions: 2D-invariant frequency-and-time convolutions substantially improve noisy-data results and provide a smaller benefit on clean data, whereas multiple 1D-invariant layers offer only a very small benefit.The experiments use one to three same-convolution layers, optionally strided across frequency or time.
  • 3.6 Striding: Striding reduces time-steps and downstream computation, but it can lower accuracy, especially in English when too few time-steps remain for character outputs.Mandarin models use straightforward striding, while English models enrich the alphabet with word, syllable, or n-gram symbols to mitigate the problem.
  • 3.6 Striding: Non-overlapping bigrams allow larger strides without sacrificing WER, reducing the unrolled RNN’s time-steps and benefiting computation and memory usage.Bigrams shorten the output transcription by grouping adjacent characters, with unigrams retained where necessary.

3.7 Row Convolution and Unidirectional Models · 3.8 Language Model · 3.9 Adaptation to Mandarin

The paper enables low-latency streaming with a unidirectional RNN architecture augmented by row convolution, while external language models improve recognition and Mandarin adaptation directly outputs Chinese characters. Mandarin requires character-level modeling and language-specific architectural adjustments, including a roughly 6000-character output layer and smaller decoding beams.

  • 3.7 Row Convolution and Unidirectional Models: Unidirectional RNNs perform as well as bidirectional models while enabling forward-only, online transcription as utterances stream.Bidirectional models require the entire sample, making them difficult to deploy with low latency.
  • 3.7 Row Convolution and Unidirectional Models: Row convolution uses a small future context of τ steps to combine the feature matrix ht:t+τ at each time-step.The layer defines a parameter matrix W with the same size as the d × (τ + 1) feature matrix.
  • 3.7 Row Convolution and Unidirectional Models: Placing row convolution above all recurrent layers supports finer-granularity streaming and yields better Mandarin CER than the best bidirectional model.The authors conjecture that recurrent layers learn useful representations, leaving row convolution to gather information for the classifier.
  • 3.8 Language Model: The system combines CTC-network and language-model log probabilities with an insertion term, tuning their weights on development data and decoding with beam search.The language-model weight α controls relative contributions, while β encourages more words in the transcription.
  • 3.8 Language Model: External language models improve both English WER and Mandarin CER, although relative gains decline from 48% to 36% in English and 27% to 23% in Mandarin as networks deepen.The authors hypothesize that additional recurrent layers build a stronger implicit language model.
  • 3.9 Adaptation to Mandarin: Mandarin recognition can output Chinese characters directly, avoiding an explicit pronunciation model and explicit modeling of language-specific pronunciation features such as tones.This provides an end-to-end Mandarin system without constructing a pronunciation model for language porting.
  • 3.9 Adaptation to Mandarin: Mandarin uses a character-level language model with character insertion, and beam size 200 causes negligible CER degradation because decoding performance levels off at smaller beams.The network output layer contains about 6000 characters, including the Roman alphabet; the test set has only 0.74% out-of-vocabulary characters.

4 System Optimizations

The system combines optimized software, synchronous multi-GPU training, and specialized GPU implementations to accelerate scalable Deep Speech 2 training. These optimizations improve scaling, reduce training time, and simplify memory management.

  • Optimized training system: Tens of millions of parameters and tens of single-precision exaFLOPs motivated a highly optimized C++/CUDA training system.The system combines a C++ deep learning library with a high-performance linear algebra library in CUDA and C++.
  • Multi-GPU scaling: Near-linear weak scaling makes one-epoch training time halve when the number of GPUs doubles.The experiment keeps the minibatch per GPU at 64, while typical training uses 8 or 16 GPUs with minibatches of 512 or 1024.
  • Communication optimization: 2.5× faster full-run training results from the custom all-reduce implementation compared with using OpenMPI directly on 8 or 16 GPUs.The ring implementation avoids extraneous CPU-GPU copies and supports GPUDirect through OpenMPI’s smcuda transport.
  • GPU CTC computation: 95 minutes per English epoch and 25 minutes per Mandarin epoch are saved by the GPU CTC implementation.The GPU implementation reduces overall training time by 10-20% through refactored dependencies and optimized parallel sorting.
  • Memory management: Over 2x slowdown from std::malloc in some cases motivated fast GPU and CPU allocation with fallback beyond available GPU memory.The fallback mechanism makes the system simpler, more robust, and more efficient when allocations exceed GPU capacity.

5 Training Data

Deep Speech 2 trains English and Mandarin models on extensive labeled speech corpora, with alignment, segmentation, filtering, and noise augmentation used to make noisy internal data suitable for training. Increasing labeled data reduces WER following a power law, while broader speech contexts may be equally important for continued improvement.

  • Dataset scale: 11,940 hours of English speech contain 8 million labeled utterances, while 9,400 hours of Mandarin audio contain 11 million labeled utterances.The corpora are substantially larger than those commonly reported in speech recognition literature.
  • Data preparation: An alignment, segmentation, and filtering pipeline converts long, noisy clips into shorter utterances with fewer erroneous transcriptions.It uses bidirectional RNN-CTC alignment, segments on consecutive blanks, and filters examples using predicted transcription quality.
  • Data augmentation: Noise augmentation increases the effective training-data size and robustness to noisy speech, but excessive augmentation can hinder optimization and worsen results.The authors tune augmentation to balance robustness against optimization difficulty.
  • Scaling with data: WER decreases with a power law as labeled training data increases for both regular and noisy development sets.Models were trained for up to 20 epochs and usually early-stopped using a held-out development set.
  • Scaling with data: The authors hypothesize that expanding speech contexts—including speakers, noise, environments, and microphones—may matter as much as increasing raw hours.They note that the necessary labels were unavailable to validate this hypothesis.

6 Results

Across challenging English and Mandarin speech conditions, DS2 improves substantially with scale and architecture, outperforming or matching humans on several benchmarks while remaining behind humans on most accented-speech categories. The best Mandarin system also slightly surpasses typical human performance on short voice-query utterances.

  • English speech: At 100 million parameters, the simple RNN outperforms the GRU and trains faster despite having two additional convolutional layers.This contrasts with the diminishing GRU advantage observed as model size increases.
  • English speech: 100 million parameters yields the lowest English generalization errors, with performance improving consistently up to that model size.All subsequent English DS2 results use this 100-million-parameter RNN model.
  • English speech: 43.4% relative improvement: the 100-million-parameter DS2 RNN outperforms DS1 on a 3,300-utterance internal set spanning accents, noise, spontaneous, and conversational speech.The test set includes low signal-to-noise speech from far-field or background noise.
  • Read speech: DS2 outperforms humans on 3 of 4 clean read-speech test sets and is competitive on the fourth.The benchmarks use Wall Street Journal and LibriSpeech corpora.
  • Accented speech: With more accented training data and a suitable architecture, performance improves across all accent groups, but humans remain better for every group except Indian.The accented-speech evaluation uses VoxForge, grouped into four accent categories.
  • Mandarin speech: 3.7% versus 4.0%: the best Mandarin system achieves a lower error rate than a group of five humans on 100 short voice-query utterances.The comparison used randomly selected utterances labeled jointly by the human group.

7 Deployment

The paper modifies the network, normalization, decoding, and deployment computation to achieve low-latency, high-throughput speech recognition. Batch Dispatch, half-precision arithmetic, specialized kernels, and beam-search pruning enable deployment with only a 5% relative character-error-rate degradation.

  • Batch Dispatch: Batch Dispatch assembles user-request streams into batches before forward propagation, trading larger-batch efficiency against increased waiting time.An eager scheduler processes each batch immediately after the previous one, reducing end-user latency without maximizing batch size.
  • Batch Dispatch: 44 ms median and 70 ms 98 percentile latency are achieved with 10 concurrent streams, while batching shifts work into more efficient batches as server load increases.These results show that large models can be deployed at high throughput and low latency.
  • Numerical efficiency: 16-bit floating-point deployment does not measurably change recognition accuracy and reduces memory and bandwidth costs for RNN weight caching and streaming.Deployment is less sensitive to numerical precision than training because network weights are not updated.
  • Numerical efficiency: A custom half-precision matrix-matrix multiplication kernel sustains higher throughput than Nervana kernels across deployment batch sizes, and both outperform NVIDIA CUBLAS 7.0.The custom kernel targets the small-batch, bandwidth-bound regime common during deployment.
  • Deployment results: The deployed system achieves 6.10 character error rate versus 5.81 for the research system, a 5% relative degradation on 2,000 held-out utterances.The deployed model uses five forward-only recurrent layers, 16-bit precision, batching, and reduced beam-search cost.

8 Conclusion

The conclusion presents end-to-end deep learning as a scalable approach that has narrowed Deep Speech’s human-performance gap, transfers quickly across languages, and may eventually surpass humans broadly. These results were enabled by effective architectural and optimization techniques supported by high-performance computing.

  • Conclusion: Deep Speech significantly narrowed its transcription-performance gap with human workers by leveraging more data and larger models.The authors argue that end-to-end systems can improve continually as data and computation increase.
  • Conclusion: The approach transferred quickly to new languages because it is highly generic.The conclusion explicitly identifies rapid application to new languages as a benefit of the generic approach.
  • Conclusion: SortaGrad, Batch Normalization, larger-stride RNNs with English bigram outputs, and bidirectional or unidirectional model searches were effective techniques.These techniques were explored as part of the architecture and optimization work underlying the reported results.
  • Conclusion: The authors conclude that end-to-end deep learning techniques will continue to scale toward a single speech system that outperforms humans in most scenarios.They base this vision on results showing that gaps from human performance have fallen rapidly where systems are not already comparable.

A Scalability improvements

This section examines the paper’s scalability improvements in greater detail.

  • A Scalability improvements: The section focuses on scalability improvements.It provides a more detailed discussion of these improvements.
  • A Scalability improvements: The authors discuss scalability improvements rather than introducing a separate topic.The passage identifies the section’s subject as scalability improvements.
  • A Scalability improvements: The section presents further detail about the paper’s approach to scalability.Its stated purpose is to discuss the scalability improvements in more detail.

A.1 Node and cluster architecture

Each compute-dense training node combines 2 Intel CPUs, 8 NVIDIA Titan X GPUs, 53 teraFLOP/second peak throughput, 384 GB CPU memory, and 8 TB of RAID-0 storage. The architecture maximizes GPU communication within PCI root complexes using GPUDirect, while FDR Infiniband connects cluster nodes for gradient transfer.

  • Node architecture: 53 teraFLOP/second peak single-precision throughput comes from a node with 2 Intel CPUs and 8 NVIDIA Titan X GPUs.Each node also provides 384 GB of CPU memory and 8 TB of storage built from two 4 TB hard disks in RAID-0.
  • Node architecture: Maximizing GPUs within the same PCI root complex enables faster GPUDirect communication for transferring gradient matrices between GPUs.The node schematic identifies devices sharing a PCI root complex and uses this arrangement for efficient communication.
  • Cluster architecture: FDR Infiniband connects all cluster nodes and is primarily used to transfer gradients during back-propagation.

A.2 GPU Implementation of CTC Loss Function

The GPU implementation of CTC eliminates costly CPU–GPU transfers and inefficient CPU parallelism by computing matrix columns in parallel and using GPU reductions for gradients. It preserves training effectiveness despite floating-point differences from the CPU implementation.

  • Motivation: The CPU implementation is inefficient because it transfers large CTC matrices between GPU and CPU, especially for Mandarin, and does not exploit SIMD parallelism effectively.The α matrix has S = 2(L + 1) rows and T columns; Mandarin matrices can contain hundreds of millions of entries.
  • GPU computation: The GPU implementation computes every α-matrix element in each column, assigning each utterance to a CUDA thread block and parallelizing independent within-column computations.Dependencies remain between consecutive time-step columns, and the backward pass reverses this ordering for β.
  • Gradient computation: Gradient computation uses character-keyed reductions and key-value sorting to aggregate α-plus-β values despite repeated characters and inserted blank symbols.The implementation uses the ModernGPU library for reduction and sorts character keys with their utterance-label indices.
  • Memory optimization: Shared memory and registers accelerate the kernels, while backward computation stores only one β column at a time as gradients are formed.The full α matrix fits in shared memory, but limited shared memory prevents storing the entire β matrix.
  • Numerical behavior: Floating-point inaccuracies make GPU and CPU results non-bitwise-identical, but both implementations train models equally well when utterances are sorted by length.The discrepancy is attributed especially to inaccuracies in transcendental floating-point functions.
Loading 1512.02595v1…