Source-linked AI summary
SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition
Daniel S. Park, William Chan, Yu Zhang, Chung-Cheng Chiu, Barret Zoph, Ekin D. Cubuk, Quoc V. Le
TL;DR
ASR networks can overfit easily and require large amounts of training data. This paper introduces SpecAugment, which warps and masks log mel spectrograms during training, achieving state-of-the-art results on LibriSpeech and Switchboard, including 6.8% WER on LibriSpeech test-other without a language model.
Problem
ASR models tend to overfit easily and require large amounts of training data.
Method
SpecAugment applies time warping and block masking of time steps and mel-frequency channels directly to log mel spectrograms during training.
Results
6.8% WER on LibriSpeech test-other without a language model, with state-of-the-art results on LibriSpeech 960h and Switchboard 300h.
Takeaways & Limitations
SpecAugment enables end-to-end LAS networks to surpass hybrid systems even without a language model.
Takeaways & Limitations
Time warping contributes little relative to the other augmentations and should be dropped first under budgetary limitations.
Abstract
from arXiv · showhide
We present SpecAugment, a simple data augmentation method for speech recognition. SpecAugment is applied directly to the feature inputs of a neural network (i.e., filter bank coefficients). The augmentation policy consists of warping the features, masking blocks of frequency channels, and masking blocks of time steps. We apply SpecAugment on Listen, Attend and Spell networks for end-to-end speech recognition tasks. We achieve state-of-the-art performance on the LibriSpeech 960h and Swichboard 300h tasks, outperforming all prior work. On LibriSpeech, we achieve 6.8% WER on test-other without the use of a language model, and 5.8% WER with shallow fusion with a language model. This compares to the previous state-of-the-art hybrid system of 7.5% WER. For Switchboard, we achieve 7.2%/14.6% on the Switchboard/CallHome portion of the Hub5'00 test set without the use of a language model, and 6.8%/14.1% with shallow fusion, which compares to the previous state-of-the-art hybrid system at 8.3%/17.3% WER.
1. Introduction
SpecAugment addresses ASR overfitting and data demands by applying a simple, computationally cheap augmentation directly to log mel spectrograms. Its time warping and time/frequency masking enable LAS networks to achieve state-of-the-art results without language models.
- ASR models often overfit easily and require large amounts of training data.
- SpecAugment operates directly on the log mel spectrogram rather than raw audio, making it simple and computationally cheap to apply.
- SpecAugment combines time warping with masking blocks of consecutive time steps or mel frequency channels.
- 2.8% WER on LibriSpeech test-clean and 6.8% WER on test-other were achieved without an LM using LAS networks.
2. Augmentation Policy
SpecAugment applies time warping, frequency masking, and time masking directly to log mel spectrograms to improve robustness to temporal deformations and partial information loss. The authors also consider multiple overlapping masks through four hand-crafted policies: LB, LD, SM, and SS.
- Policy objectives: The policy acts directly on log mel spectrograms to promote robustness to time deformations, partial frequency loss, and partial loss of short speech segments.The masked inputs are intended to help the network learn useful features.
- Time warping: Time warping randomly shifts a central horizontal spectrogram point left or right by a uniformly sampled distance up to the time-warp parameter W.The spectrogram is treated as an image with time horizontal and frequency vertical, using six fixed anchor points.
- Frequency masking: Frequency masking removes f consecutive mel frequency channels, with f sampled up to F and the starting channel selected within the available frequency range.The masked interval is [f0, f0 + f), and ν denotes the number of mel frequency channels.
- Time masking: Time masking removes t consecutive time steps, with t sampled up to T and the start selected within the available sequence; its width is additionally bounded by p times the sequence length.The masked interval is [t0, t0 + t).
- Combined policies: The authors allow multiple, potentially overlapping frequency and time masks and study four hand-crafted policies: LibriSpeech basic (LB), LibriSpeech double (LD), Switchboard mild (SM), and Switchboard strong (SS).The policies’ parameters are summarized in Table 1.
3. Model
The ASR systems use LAS networks with a CNN front end, stacked bidirectional LSTM encoder, attention, and an RNN decoder. Training relies on parameterized learning-rate schedules, weight noise, label smoothing, and optional shallow-fusion language models.
- LAS architecture: LAS-d-w models process log mel spectrograms through a 2-layer CNN, d stacked bidirectional LSTMs, attention vectors, and a 2-layer RNN decoder.The CNN uses max-pooling and stride 2; the encoder LSTMs have cell size w, and the decoder has cell dimension w.
- Training schedules: Longer schedules improve final network performance, particularly when augmentation is present, motivating very long schedules for maximizing performance.The schedules are used both to verify the benefit of longer training and to maximize network performance.
- Training schedules: Learning-rate schedules ramp up, hold, then exponentially decay the rate to 1/100 of its maximum, using timestamps (sr, si, sf).The rate remains constant after reaching 1/100 of its maximum value.
- Training schedules: Training adds variational weight noise with standard deviation 0.075 during the learning-rate high plateau and uniform label smoothing with uncertainty 0.1.Weight noise begins at snoise and remains constant; label smoothing assigns confidence 0.9 to the correct class and may be disabled when decay starts.
- Language-model fusion: Shallow fusion incorporates an RNN language model and coverage penalty c by jointly scoring the next token with the ASR and language models.For LibriSpeech, the language model is a 2-layer RNN with embedding dimension 1024; for Switchboard, it is a 2-layer RNN with embedding dimension 256.
4. Experiments
Experiments on LibriSpeech and Switchboard show that SpecAugment achieves state-of-the-art performance against heavily engineered hybrid systems. The studies evaluate augmentation policies across standardized filter-bank setups, training schedules, label smoothing, and shallow-fusion language models.
- SpecAugment experiments on LibriSpeech and Switchboard report state-of-the-art results that outperform heavily engineered hybrid systems.
- LibriSpeech: LibriSpeech uses 80-dimensional filter banks with delta and delta-delta acceleration and a 16k word piece model.
- LibriSpeech: LAS-6-1280 achieves state-of-the-art LibriSpeech performance without a language model, using policy LD and schedule L.The model is trained for approximately 24 days, with label smoothing enabled for time steps below 140k.
- LibriSpeech: Shallow fusion with a language model further improves LibriSpeech performance.
- Switchboard: Switchboard uses 80-dimensional filter banks with delta and delta-delta acceleration and a 1k word-piece model built from combined Switchboard and Fisher vocabularies.
- Switchboard: Label smoothing and augmentation have an additive effect on Switchboard performance.LAS-6-1280 with schedule L is trained for approximately 24 days to obtain state-of-the-art performance, with label smoothing enabled throughout training.
5. Discussion
The discussion finds that augmentation shifts training from overfitting toward underfitting, which can be addressed with larger networks and longer training. Time warping contributes only modestly and is the first augmentation to drop under budget constraints, while label smoothing can destabilize augmented training.
- Time warping has a small but existent effect and is the least influential, most expensive augmentation.It should therefore be the first augmentation dropped when budget limitations arise.
- Label smoothing increases instability in augmented LibriSpeech training, especially during learning-rate decay.The authors address this with a schedule that smooths labels only during the initial learning-rate phases.
- Augmentation converts overfitting into underfitting on both augmented and unaugmented training data.This underfitting is identified as the major benefit of training with augmentation.
- Larger networks and longer training substantially improve performance by alleviating augmentation-induced underfitting.The reported performance resulted from iteratively applying harsh augmentation, then widening and deepening networks and extending training schedules.
- SpecAugment frequency masks differ from a related CNN method because their size and position are stochastic for every minibatch input.The related method pre-groups adjacent frequencies into bins and randomly zeros those bins per minibatch.
6. Conclusions
SpecAugment substantially improves end-to-end LAS ASR, achieving state-of-the-art results on LibriSpeech 960h and Switchboard 300h while surpassing hybrid systems without a language model. By reducing overfitting, it enables further gains from larger networks and longer training.
- Conclusions: SpecAugment greatly improves ASR network performance, achieving state-of-the-art results on LibriSpeech 960h and Switchboard 300h with end-to-end LAS networks.The method augments the training set using simple hand-crafted policies.
- Conclusions: SpecAugment surpasses hybrid systems even without the aid of a language model.The augmentation policies use simple hand-crafted training-set transformations.
- Conclusions: SpecAugment converts ASR from an over-fitting to an under-fitting problem, enabling gains from bigger networks and longer training.These additional gains follow the shift in the limiting training behavior caused by augmentation.