Source-linked AI summary
Improved Regularization Techniques for End-to-End Speech Recognition
Yingbo Zhou, Caiming Xiong, Richard Socher
TL;DR
Flexible end-to-end speech models can overfit, yet data augmentation and dropout have been relatively underexplored for them. This paper perturbs audio and applies dropout to inputs across network layers, reporting over 20% relative performance improvement on WSJ and LibriSpeech. The reported regularization approach also avoids recurrent dropout because of its implementation and computation-time trade-off.
Problem
Method
The paper perturbs raw audio in tempo, pitch, volume, temporal alignment, and noise, and applies dropout to inputs of all network layers.
Source: §1 · p. 1Results
Over 20% relative performance improvement was reported on WSJ and LibriSpeech using data augmentation and dropout.
Source: §1 · p. 1Takeaways & Limitations
For these end-to-end speech models, regularization addresses overfitting risk associated with model flexibility and supports generalization.
Source: §1 · p. 1, §4 · p. 2Recurrent dropout was not used because it was not a cuDNN drop-in substitution and switching implementations would significantly increase RNN computation time.
Source: §5 · p. 3
Abstract
from arXiv · showhide
Regularization is important for end-to-end speech models, since the models are highly flexible and easy to overfit. Data augmentation and dropout has been important for improving end-to-end models in other domains. However, they are relatively under explored for end-to-end speech models. Therefore, we investigate the effectiveness of both methods for end-to-end trainable, deep speech recognition models. We augment audio data through random perturbations of tempo, pitch, volume, temporal alignment, and adding random noise.We further investigate the effect of dropout when applied to the inputs of all layers of the network. We show that the combination of data augmentation and dropout give a relative performance improvement on both Wall Street Journal (WSJ) and LibriSpeech dataset of over 20%. Our model performance is also competitive with other end-to-end speech models on both datasets.
1. INTRODUCTION
Because flexible end-to-end speech models can overfit, this paper investigates data augmentation and dropout as regularizers. It perturbs audio and applies dropout to layer inputs, reporting over 20% relative performance improvement on WSJ and LibriSpeech.
1. INTRODUCTION
- Data augmentation and dropout are investigated because flexible deep neural networks can overfit, while these methods regularize models in other pattern-recognition settings.The introduction describes dropout as randomly zeroing hidden-layer inputs to reduce co-adaptation among hidden units.Source: §1 · p. 1
- Audio augmentation independently changes tempo and pitch, perturbs volume and temporal alignment, and adds random noise.Dropout is also applied to each input layer of the network.Source: §1 · p. 1
- Over 20% relative performance improvement was obtained on both WSJ and LibriSpeech using data augmentation and dropout.The audio perturbations include tempo, pitch, volume, temporal alignment, and added random noise; dropout is applied to each input layer.Source: §1 · p. 1
2. RELATED WORK
Prior speech-recognition work includes feature- and audio-level augmentation, noise addition, and dropout in acoustic models. This work distinguishes its augmentation scope and dropout placement in end-to-end models.
2. RELATED WORK
- Earlier augmentation work includes feature-level methods, speed perturbation, and background-noise addition, while this work also perturbs tempo, pitch, volume, and temporal alignment.Prior examples include Gales et al., Ko et al., and noise augmentation in [8].Source: §2 · p. 1
- Dropout had been used in acoustic models [15] and at an end-to-end model's output layer [8]; this work applies it to other end-to-end layers.The authors state that, to their knowledge, this was the first such application to other layers in end-to-end speech models.Source: §2 · p. 1
3. MODEL ARCHITECTURE
The model follows Deep Speech 2's recurrent-layer structure with modifications to its front-end time-frequency convolutions. Its convolutional, recurrent, and fully connected layers form a per-character prediction network, with dropout placements shown in Fig. 1.
3. MODEL ARCHITECTURE
- The model closely follows Deep Speech 2's recurrent-layer structure but modifies the front-end time-frequency convolution layers.The full architecture is illustrated in Fig. 1.Source: §3 · p. 1
- The convolution layers use depth-wise separable convolutions followed by 1 × 1 filters, and normal convolutions are replaced with ResNet blocks.The residual connections are described as helping gradient flow during training.Source: §3 · pp. 1–2
- Fig. 1 uses colored blocks to distinguish layer types and triangles to mark dropout immediately before the pointed layer.The figure provides a visual guide to dropout placement in the model.Source: Figure 1 · p. 2
- The model uses one large-filter convolution layer, five residual convolution blocks, a four-layer bidirectional GRU network, and two fully connected layers for per-character output.Batch normalization is applied to all layers.Source: §3 · p. 2
4. PREVENTING OVERFITTING
The section describes raw-audio augmentation and dropout as regularizers for an over- five-million-parameter speech model. It separates tempo from pitch to broaden audio variation and applies dropout to layer inputs, including a fixed mask across time for recurrent networks.
4. PREVENTING OVERFITTING
- With over five million parameters, the model motivates regularization to help it generalize rather than overfit.The section presents data augmentation and dropout as its primary methods for preventing overfitting.Source: §4 · p. 2
4.1. Data Augmentation
- Separating tempo and pitch in raw-audio augmentation permits combinations such as higher pitch with slower tempo, broadening variation beyond speed perturbation alone.The authors choose data-level augmentation because raw audio has no feature-level dependencies, making it more flexible.Source: §4.1 · p. 2
- Random white noise, volume changes, and shifts of less than 10 ms add further audio variation for synthetic training data.These perturbations are combined with tempo and pitch changes.Source: §4.1 · p. 2
4.2. Dropout
- For recurrent networks, the dropout variant uses the same random input mask across time rather than resampling dimensions at each time step.The passage notes that ordinary dropout has little success when applied to recurrent networks; the fixed-mask variant is attributed to Gal and Ghahramani.Source: §4.2 · p. 2
- The section applies the modified dropout to convolutional and recurrent layer inputs, while using standard dropout on fully connected layers.The dropout mask zeros a subset of layer inputs during training; at test time, inputs are rescaled by 1−p to preserve expected pre-activation.Source: §4.2 · pp. 2–3
5. EXPERIMENTS
Experiments assess regularization on WSJ and LibriSpeech, then compare the model with other end-to-end speech systems.
5. EXPERIMENTS
- Experiments use normalized 20-ms-window, 10-ms-step spectrograms and end-to-end CTC training; recurrent-connection dropout is omitted to retain cuDNN and avoid significant RNN slowdown.The model is evaluated on WSJ and LibriSpeech.Source: §5 · p. 3
5.1. Effect of individual regularizer
- 6.42% final WER is achieved with all regularization on WSJ; LibriSpeech gains exceed 22% on test-clean and 32% on test-other.The reported LibriSpeech improvements are relative to baseline.Source: Table 1 · p. 3, Table 2 · p. 4, §5.1 · p. 4
- 22% relative improvement is reported for dropout on WSJ, while combined augmentation yields 20% over baseline; noise and tempo augmentation also improve over baseline.The tempo-based augmentation slightly outperforms speed perturbation, which yields 7.21% WER and 13.96% relative improvement.Source: §5.1 · p. 3, Table 1 · p. 3
- Regularization narrows the WSJ training–validation loss gap and lowers validation loss.Fig. 2 contrasts the baseline learning curve with the regularized model’s losses.Source: Figure 2 · p. 4, §5.1 · p. 4
5.2. Comparison to other methods
- The model is competitive with existing end-to-end methods on WSJ and LibriSpeech, and comparable to Amodei et al. on LibriSpeech using only the provided training set.Tables 3 and 4 compare word error rates on the two datasets.Source: §5.2 · p. 4, Table 3 · p. 4, Table 4 · p. 4
6. CONCLUSION
The paper combines audio augmentation and dropout on inputs to all network layers, reporting new state-of-the-art results on WSJ and LibriSpeech. Tempo- and noise-based augmentation alone yields 15–20% relative performance improvement on both datasets.
6. CONCLUSION
- 6.26% on WSJ, 5.67% on LibriSpeech test-clean, and 15.18% on LibriSpeech test-other are the reported combined-method results.The authors describe these as new state-of-the-art results on both datasets.Source: §6 · p. 4
- Tempo- and noise-based augmentation achieves 15–20% relative performance improvement on WSJ and LibriSpeech.The augmentation varies tempo and pitch, and also changes gain, shifts audio, and adds random white noise.Source: §6 · p. 4
- Applying dropout to the inputs of all network layers also produces significant performance improvements.Source: §6 · p. 4