Source-linked AI summary
Effectiveness of self-supervised pre-training for speech recognition
Alexei Baevski, Michael Auli, Abdelrahman Mohamed
TL;DR
The paper asks whether self-supervised pretraining can reduce ASR’s dependence on transcribed speech. It compares discrete and continuous representations and directly fine-tunes BERT with CTC, finding strong performance from discrete-unit pretraining, including with 10 minutes of labels.
Problem
ASR typically requires large labeled datasets to represent the many factors contributing to speech signals.
Method
The study compares quantized and non-quantized self-supervised speech representations and directly fine-tunes pretrained BERT models on transcribed speech using CTC loss.
Results
10 minutes of labeled data achieves 16.3/25.2 WER on test-clean/other, while 10-hour fine-tuning nearly matches the best 100-hour test-clean result and reduces test-other WER by 25%.
Takeaways & Limitations
Discrete-unit discovery followed by BERT training is more accurate than directly learning representations from continuous audio data.
Abstract
from arXiv · showhide
We compare self-supervised representation learning algorithms which either explicitly quantize the audio data or learn representations without quantization. We find the former to be more accurate since it builds a good vocabulary of the data through vq-wav2vec [1] to enable learning of effective representations in subsequent BERT training. Different to previous work, we directly fine-tune the pre-trained BERT models on transcribed speech using a Connectionist Temporal Classification (CTC) loss instead of feeding the representations into a task-specific model. We also propose a BERT-style model learning directly from the continuous audio data and compare pre-training on raw audio to spectral features. Fine-tuning a BERT model on 10 hour of labeled Librispeech data with a vq-wav2vec vocabulary is almost as good as the best known reported system trained on 100 hours of labeled data on testclean, while achieving a 25% WER reduction on test-other. When using only 10 minutes of labeled data, WER is 25.2 on test-other and 16.3 on test-clean. This demonstrates that self-supervision can enable speech recognition systems trained on a near-zero amount of transcribed data.
1 Introduction
The paper studies self-supervised speech representation learning to reduce ASR’s dependence on large labeled datasets. It compares discrete-unit and continuous representation learning, then directly fine-tunes pretrained BERT models for recognition.
- ASR typically requires large training volumes because speech reflects factors including noise, channel, speaker, accent, emotion, topic, and language.
- Self-supervised learning has recently produced representations that generalize across downstream NLP and speech tasks.
- The study compares discrete audio units from vq-wav2vec or clustered spectral features with continuous representations learned from raw audio and spectral features.
- Unlike previous work, the pretrained BERT models are directly fine-tuned on transcribed speech using a CTC loss rather than passed into task-specific architectures.
- 10 minutes of labeled data yields 16.3/25.2 WER on test-clean/other, while 1 hour outperforms the best reported 100-hour result on test-other.
2 Preliminaries
The preliminaries describe wav2vec’s continuous context-prediction representations and vq-wav2vec’s quantized alternative. Both use convolutional feature extraction and future-step prediction, but vq-wav2vec inserts a discrete codebook.
- BERT uses bidirectional self-attention and masked language modeling to learn representations from partially masked sequences.
- Wav2Vec: Wav2Vec encodes audio into 100 Hz representations, aggregates context, and predicts future representations against randomly sampled distractors.
- Wav2Vec: The wav2vec objective uses a step-specific affine transformation and logistic probability to distinguish the true future representation from negatives.
- Wav2Vec: The context features learned by wav2vec are reported to outperform standard spectral features as speech-recognition representations.
- vq-wav2vec: vq-wav2vec adds a quantization module between convolutional encoding and aggregation, converting dense audio representations into discrete inputs.
- vq-wav2vec: Its quantizer replaces each dense representation with a codebook vector selected from V representations of dimension d.
3 Approach
The approach pretrains BERT on either discrete acoustic units or continuous features, then fine-tunes the transformer for ASR with a CTC projection. Continuous-input pretraining instead predicts masked features through contrastive classification.
- The vq-wav2vec setup quantizes Librispeech audio into 13.5k unique codes before BERT pretraining.
- MFCC and FBANK features are quantized with k-means using 13.5k centroids to match the vq-wav2vec setup.
- Discrete BERT: Discrete inputs are trained with BERT’s masked-language-model objective, masking token spans and predicting the original tokens.
- Continuous BERT: Continuous BERT masks wav2vec, MFCC, or FBANK features and classifies the true masked feature against randomly sampled negatives.
- ASR fine-tuning: The pretrained transformer is fine-tuned by adding a projection over 29 character tokens plus a word-boundary token and minimizing CTC loss.
- ASR fine-tuning: Fine-tuning uses SpecAugment-inspired time and channel masking, with dropout additionally applied for the 10-minute and 1-hour settings.
4 Experiments
The experiments compare discrete and continuous self-supervised pre-training across limited labeled-data settings. Discrete BERT with vq-wav2vec inputs performs best, while additional BERT pre-training substantially improves wav2vec-based continuous models.
- Discrete BERT outperforms Continuous BERT in all evaluated labeled-data settings.The comparison covers discretized and continuous inputs across Librispeech clean and other subsets.
- vq-wav2vec provides about 40% relative error reduction on both test subsets compared with clustered spectral features.The gains are larger on the noisy test-other subset and hold across training-set sizes.
- 16.3/25.2 WER on test-clean/other is achieved with only 10 minutes of labeled data using Discrete BERT with vq-wav2vec inputs.Reducing labeled data from 100h to 10h increases WER by only 1.4 on test-clean and 2 on test-other for this setup.
- 10 hours of labeled data nearly matches the best reported 100-hour result on test-clean and achieves a 25% relative WER reduction on test-other.With the train-clean-100 subset, the method improves over the cited result by 1.3 WER on test-clean and 6.5 WER on test-other.
- A second representation-learning step more than halves WER for Continuous BERT with wav2vec features compared with wav2vec alone.The comparison uses 10 hours of labeled data and reports larger gains on the clean subset.
5 Discussion and Related Work
The work builds on efforts to reduce labeled-data dependence in speech recognition through self-supervised representation learning, unsupervised unit discovery, transfer learning, and semi-supervised learning.
- Self-supervised speech and language representation learning has motivated approaches that generalize across downstream tasks.
- Prior speech methods learn representations by predicting masked discrete or continuous inputs or contrastively comparing neighboring or similar-sounding segments.
- This work is inspired by efforts to reduce labeled-data dependence through unsupervised unit discovery, acoustic representation learning, cross-lingual transfer, and semi-supervised learning.
6 Conclusion and Future work
The paper systematically compares self-supervised pre-training approaches for speech recognition and identifies discrete vocabulary learning with vq-wav2vec followed by BERT training as the most effective method. Direct fine-tuning of the resulting BERT model achieves strong accuracy with substantially less labeled data, including competitive results using only 10 minutes.
- Discrete vocabulary learning with vq-wav2vec followed by BERT training is the most effective approach and outperforms direct learning from continuous audio.
- The resulting BERT model is directly fine-tuned on transcribed speech rather than feeding representations into a task-specific ASR model.
- With two orders of magnitude less labeled data, the approach achieves better test-other accuracy than the best known result trained with 100 hours.
- 10 minutes of labeled data yields WER 25.2 on test-other and WER 16.3 on test-clean.