Source-linked AI summary
wav2vec: Unsupervised Pre-training for Speech Recognition
Steffen Schneider, Alexei Baevski, Ronan Collobert, Michael Auli
TL;DR
Speech recognition typically needs substantial transcribed audio, motivating methods that exploit easier-to-collect unlabeled speech. wav2vec pre-trains convolutional raw-audio representations with a contrastive future-sample prediction objective, then improves supervised acoustic modeling. It reduces WSJ WER by up to 36% in a low-resource setting and reaches 2.43% WER, outperforming the best reported character-based result with far less labeled data.
Problem
Speech recognition systems require large amounts of transcribed audio, while obtaining labeled speech is costly and limited.
Method
wav2vec pre-trains convolutional representations of raw audio by distinguishing true future latent samples from distractors, then uses them in supervised acoustic models.
Results
2.43% WER on WSJ outperforms the best reported character-based system while using two orders of magnitude less transcribed training data.
Takeaways & Limitations
Pre-training on unlabeled audio improves speech recognition both with scarce transcribed data and when all WSJ training data is used.
Takeaways & Limitations
Sampling negatives from different sequences and speakers yields inferior results.
Abstract
from arXiv · showhide
We explore unsupervised pre-training for speech recognition by learning representations of raw audio. wav2vec is trained on large amounts of unlabeled audio data and the resulting representations are then used to improve acoustic model training. We pre-train a simple multi-layer convolutional neural network optimized via a noise contrastive binary classification task. Our experiments on WSJ reduce WER of a strong character-based log-mel filterbank baseline by up to 36% when only a few hours of transcribed data is available. Our approach achieves 2.43% WER on the nov92 test set. This outperforms Deep Speech 2, the best reported character-based system in the literature while using two orders of magnitude less labeled training data.
1 INTRODUCTION
wav2vec applies unsupervised pre-training to raw audio so learned representations can improve supervised speech recognition when transcribed data is scarce. It uses a convolutional model and substantially improves WSJ performance, including a 2.43% WER result with much less labeled data.
- wav2vec learns raw-audio representations through unsupervised pre-training and uses them to improve supervised speech recognition.
- The approach exploits unlabeled audio, which is easier to collect than labeled speech data, for downstream acoustic modeling.
- Up to 36% WER reduction is achieved with only about eight hours of transcribed audio.
2 PRE-TRAINING APPROACH
wav2vec encodes raw audio into low-frequency features and contextual representations with stacked convolutional networks. It trains these representations by distinguishing true future latent samples from distractors using a contrastive objective, then supplies the context representations to an acoustic model.
- The encoder maps raw audio to low-frequency representations, while the context network combines multiple time steps into contextualized tensors.The encoder produces representations every 10 ms, and the context network uses a roughly 210 ms receptive field.
- Normalization is chosen to be invariant to input scaling and offset, supporting representations that generalize across datasets.
- The larger wav2vec variant increases model capacity with a deeper context network, skip connections, and an approximately 810 ms receptive field.
- The contrastive objective distinguishes a true future sample from distractors drawn from a proposal distribution.The loss is summed across prediction steps, with ten negative examples sampled in practice.
- After pre-training, context-network representations replace log-mel filterbank features as acoustic-model inputs.
3 EXPERIMENTAL SETUP
The experiments evaluate wav2vec pre-training across TIMIT, WSJ, and Librispeech settings, using acoustic-model baselines, language-model decoding, and varied unlabeled-data configurations.
- Datasets: The evaluation covers TIMIT, WSJ, and Librispeech, with WSJ tested on si284, nov93dev, and nov92 splits.WSJ contains about 81 hours of transcribed audio, while Librispeech provides 960 hours for pre-training.
- Acoustic models: The WSJ baseline uses wav2letter++ with log-mel filterbanks, while pre-trained embeddings replace those input features in the acoustic model.Table 1 evaluates test and validation performance using LER and WER.
- Decoding: Decoding combines acoustic-model emissions with a lexicon and separate language models, including a 4-gram KenLM model and convolutional alternatives.The decoder uses beam search and tunes language-model, word-penalty, and silence-penalty weights by random search.
- Pre-training models: Pre-training experiments vary the unlabeled audio source, including WSJ and Librispeech, with ten negatives sampled for each of 12 prediction tasks.The pre-training models use Adam with cosine learning-rate schedules.
- Pre-training models: Audio sequences are randomly cropped to at most 150k frames, providing data augmentation while standardizing batch lengths across GPUs.Cropping removes speech from either sequence end and removes on average 25% of training data.
4 RESULTS
Pre-training improves speech-recognition performance across WSJ and TIMIT, with larger unlabeled datasets helping and especially strong gains in low-resource WSJ settings.
- 4.1 PRE-TRAINING FOR THE WSJ BENCHMARK: 36% lower WER is achieved on nov92 with only about eight hours of transcribed data using full-Librispeech pre-training.Pre-training on WSJ audio alone performs worse than pre-training on the larger Librispeech corpus.
- 4.2 PRE-TRAINING FOR TIMIT: TIMIT phoneme-recognition accuracy matches the state of the art, with the best result obtained from the largest pre-training dataset.Accuracy increases steadily as the amount of pre-training data grows.
- 4.3 ABLATIONS: More than ten negative samples do not improve performance, while increasing their number increases training time.This ablation keeps other experimental factors fixed.
- 4.3 ABLATIONS: A 150k-frame crop gives the best performance, whereas unrestricted sequence length gives the worst accuracy.The unrestricted setting likely provides the least data augmentation.
- 4.3 ABLATIONS: Predicting more than 12 future steps does not improve performance and increases training time.
5 CONCLUSIONS
The paper introduces wav2vec as a fully convolutional approach to unsupervised speech-recognition pre-training. It achieves 2.43% WER on WSJ while using two orders of magnitude less transcribed training data, and benefits from more pre-training data.
- 5 CONCLUSIONS: 2.43% WER on WSJ outperforms the next-best known character-based speech-recognition model while using two orders of magnitude less transcribed data.
- 5 CONCLUSIONS: More unlabeled pre-training data improves performance across the evaluated settings.
- 5 CONCLUSIONS: wav2vec improves both resource-poor setups and settings using all WSJ training data.