Source-linked AI summary
SpecAugment on Large Scale Datasets
Daniel S. Park, Yu Zhang, Chung-Cheng Chiu, Youzheng Chen, Bo Li, William Chan, Quoc V. Le, Yonghui Wu
TL;DR
The paper asks whether SpecAugment remains effective for large-scale, multi-domain ASR and develops length-adaptive time masking for variable-length utterances. It evaluates SpecAugment against MTR on the Google Multidomain Dataset and adaptive policies on LibriSpeech, improving performance across domains and reaching 2.2% and 5.2% WER on LibriSpeech test-clean and test-other. Adaptive masking, however, does not outperform fixed masking on the Google Multidomain Dataset.
Problem
The paper investigates whether SpecAugment remains effective for large-scale ASR tasks with multiple domains and varied utterance lengths.
Method
The paper applies SpecAugment to the Google Multidomain Dataset, mixes SpecAugmented and MTR data, and introduces time masking whose size or multiplicity depends on utterance length.
Results
SpecAugment mixed with MTR improves performance across all domains, while adaptive masking reaches 2.2% WER on test-clean and 5.2% WER on test-other for LibriSpeech.
Takeaways & Limitations
SpecAugment is presented as a scalable alternative to more resource-intensive augmentation methods for large-scale ASR pipelines.
Takeaways & Limitations
No adaptive policy tested outperforms fixed-policy SpecAugment on the Google Multidomain Dataset.
Abstract
from arXiv · showhide
Recently, SpecAugment, an augmentation scheme for automatic speech recognition that acts directly on the spectrogram of input utterances, has shown to be highly effective in enhancing the performance of end-to-end networks on public datasets. In this paper, we demonstrate its effectiveness on tasks with large scale datasets by investigating its application to the Google Multidomain Dataset (Narayanan et al., 2018). We achieve improvement across all test domains by mixing raw training data augmented with SpecAugment and noise-perturbed training data when training the acoustic model. We also introduce a modification of SpecAugment that adapts the time mask size and/or multiplicity depending on the length of the utterance, which can potentially benefit large scale tasks. By using adaptive masking, we are able to further improve the performance of the Listen, Attend and Spell model on LibriSpeech to 2.2% WER on test-clean and 5.2% WER on test-other.
1. INTRODUCTION
The paper tests whether SpecAugment remains effective at large scale, especially across multiple domains, and compares it with noise-based augmentation. It also introduces adaptive time masking and reports gains on LibriSpeech.
- Motivation: SpecAugment is evaluated on the large-scale, multi-domain Google Multidomain Dataset to test whether its prior ASR benefits persist at scale.The dataset contains multiple test sets from disparate domains.
- Main results: Mixing SpecAugmented data with MTR data improves performance across all domains.MTR combines clean audio with noise-perturbed audio using a room simulator.
- Main results: SpecAugment applied to clean data outperforms the baseline on all natural test sets but underperforms on a synthetic MTR-based test set.Applying SpecAugment on top of MTR instead degrades performance across most domains.
- Practical significance: SpecAugment uses negligible additional computation, requires no extra audio, and can be applied online as training sets grow.These properties make it a scalable alternative to more resource-intensive augmentation methods.
- Adaptive masking: Adaptive time masking varies mask size and/or multiplicity with utterance length to address length variation in multi-domain data.The paper studies adaptive policies on the Google Multidomain Dataset and LibriSpeech 960h.
2. SPECAUGMENT AND ADAPTIVE MASKING
SpecAugment composes time warping, frequency masking, and time masking on spectrograms. The paper adapts time masking to spectrogram length by scaling mask multiplicity or size, while capping adaptive mask counts at 20.
- SpecAugment components: SpecAugment combines time warping, frequency masking, and time masking, with τ and ν denoting spectrogram time and frequency dimensions.The standard policies apply these augmentations a fixed number of times.
- Time warping: Time warping selects a displacement and start point, then maps the start point while keeping the spectrogram boundary points fixed.The warped features satisfy xwarp(W(t)) = xorig(t).
- Frequency masking: Frequency masking samples a mask size up to F and masks consecutive log-mel frequency channels.The masked interval is [f0, f0+f).
- Time masking: Time masking samples a mask size up to T and masks consecutive time steps.The masked interval is [t0, t0+t).
- Adaptive masking: Fixed time-mask counts may be too weak for long utterances or too severe for short ones, motivating length-adaptive masking.The paper considers adaptive multiplicity and adaptive size as two alternatives.
- Adaptive masking: Adaptive multiplicity sets Mt-mask = ⌊pM · τ⌋, while adaptive size sets T = ⌊pS · τ⌋.The number of adaptive time masks is capped at 20.
3.1. LibriSpeech 960h
On LibriSpeech 960h, the study compares fixed and adaptive SpecAugment policies within a LAS-based setup. The adaptive policy performs better than the fixed policy before and after language-model shallow fusion.
- Experimental setup: The LibriSpeech setup uses the LAS-6-1280 model, long training schedule, and shallow fusion with a 3-layer LSTM language model.The language model has width 4096 and dev-set word-level perplexity 63.6.
- Augmentation policies: The fixed LibriSpeech Double policy applies two frequency masks with F = 27 and two time masks with T = 100 after time warping with W = 80.
- Augmentation policies: LibriFullAdapt uses two frequency-mask applications with F = 27 and adaptive time-mask multiplicity and size, with pM = 0.04 and pS = 0.04.The adaptive time masks are applied after time warping with W = 80.
- Results: The adaptive policy performs better than the fixed policy both before and after shallow fusion with the language model.
3.2. Google Multidomain Dataset
On the Google Multidomain Dataset, the study evaluates SpecAugment across five test sets and several data configurations. Clean-data SpecAugment improves natural test sets over the baseline, while combining SpecAugmented and MTR data addresses synthetic-test degradation; adaptive masking does not outperform the fixed policy here.
- Data and evaluation: The experiment evaluates Search, Search-Noisy, TTS-Audiobook, Telephony, and YouTube test sets using anonymized training and testing data.
- Data and augmentation: The compared configurations are MTR, clean data, MTR with SpecAugment, clean data with SpecAugment, and an 8:2 mixture of SpecAugmented clean data and MTR data.Augmentation is applied after unstacking the spectrogram features and before restacking them for the acoustic model.
- Data and augmentation: SpecAugBasic uses two frequency masks and two time masks with T = 50, without time warping; a frequency-masking-only control uses two masks with F = 27.
- RNN-T: Time masking yields gains despite the RNN-T model’s weaker context from streaming operation.Gaussian noise is added to time-masked regions to stabilize training with layer normalization.
- Results: SpecAugment on top of MTR degrades performance below the baseline across all test sets.
- Results: Clean-data SpecAugBasic outperforms the baseline on all natural test sets but performs worse on the synthetic MTR-derived test set; mixing SpecAugmented and MTR data addresses this degradation.
- Adaptive masking: Adaptive time masking does not outperform fixed-policy SpecAugBasic on this dataset.The benefit of adaptive time masking on the Google Multidomain Dataset remains unobserved.
- Results: SpecAugment noticeably improves the performance of the streaming model, with time masking especially important for the YouTube evaluation.
4. SUMMARY AND DISCUSSION
The paper finds that SpecAugment offers strong gains on large-scale datasets and introduces adaptive time-masking, whose benefits are demonstrated on LibriSpeech but not the Google Multidomain Dataset.
- SpecAugment yields better gains on large-scale datasets than more sophisticated augmentation methods.
- Adaptive time-masking improves performance on LibriSpeech 960h but does not outperform non-adaptive masking on the Google Multidomain Dataset.
- Further exploration of adaptive masking may improve SpecAugment for large-scale tasks.