PaperlayerView exact revision on arXiv

Source-linked AI summary

wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations

Alexei Baevski, Henry Zhou, Abdelrahman Mohamed, Michael Auli

arXiv:2006.11477v3cs.CLcs.LGcs.SDeess.AS

TL;DR

Speech recognition typically depends on large amounts of transcribed speech, which are unavailable for most languages. wav2vec 2.0 learns representations from unlabeled audio with masked latent inputs and quantized contrastive targets, then fine-tunes on labeled speech, achieving strong recognition with as little as 10 minutes of labels.

  • Problem

    Speech recognition systems require hundreds or thousands of hours of labeled speech, limiting coverage for the many languages lacking such transcriptions.

  • Method

    wav2vec 2.0 masks latent speech representations and learns contextualized representations through contrastive prediction of jointly learned quantized speech targets before fine-tuning.

  • Results

    4.8/8.2 WER on Librispeech test-clean/other with 10 minutes of labeled data, while outperforming prior best results on the 100-hour setup using 100 times less labeled data.

  • Takeaways & Limitations

    The results demonstrate the feasibility of accurate speech recognition with very small amounts of annotated data.

  • Takeaways & Limitations

    The acoustic model uses characters while the language model uses words, a vocabulary mismatch the authors say is likely detrimental.

Abstract

from arXiv · show

We show for the first time that learning powerful representations from speech audio alone followed by fine-tuning on transcribed speech can outperform the best semi-supervised methods while being conceptually simpler. wav2vec 2.0 masks the speech input in the latent space and solves a contrastive task defined over a quantization of the latent representations which are jointly learned. Experiments using all labeled data of Librispeech achieve 1.8/3.3 WER on the clean/other test sets. When lowering the amount of labeled data to one hour, wav2vec 2.0 outperforms the previous state of the art on the 100 hour subset while using 100 times less labeled data. Using just ten minutes of labeled data and pre-training on 53k hours of unlabeled data still achieves 4.8/8.2 WER. This demonstrates the feasibility of speech recognition with limited amounts of labeled data.

1 Introduction

wav2vec 2.0 learns contextualized speech representations and discrete speech units jointly from raw unlabeled audio, then fine-tunes on labeled speech for recognition. This framework addresses limited transcription availability and achieves strong performance with only 10 minutes of labeled data.

  • Motivation: Thousands of hours of transcribed speech are typically required for acceptable recognition, but such labeled data is unavailable for most of the nearly 7,000 languages.The introduction motivates learning from abundant unlabeled speech rather than relying solely on labeled examples.
  • Background: Self-supervised learning learns general representations from unlabeled examples and fine-tunes them on labeled data.The introduction situates this paradigm as successful in natural language processing and active in computer vision.
  • Framework: The framework encodes raw speech with a convolutional network, masks latent spans, contextualizes them with a Transformer, and trains contrastively to identify the true latent.This combines latent-space masking with contextual representation learning and a contrastive objective.
  • Framework: Discrete speech units are learned with Gumbel-Softmax and used as contrastive targets, while pre-training and labeled-data fine-tuning are combined end-to-end.Fine-tuning uses a Connectionist Temporal Classification loss for downstream speech recognition.
  • Results: 4.8/8.2 WER on Librispeech clean/other test sets is achieved using only 10 minutes of labeled data.The results also report substantially better performance from jointly learned discrete units than from fixed units learned in a prior step.

2 Model

The model maps raw audio to latent speech representations with a convolutional feature encoder, contextualizes them using a Transformer, and discretizes encoder outputs for self-supervised targets. Product quantization uses multiple codebooks, while differentiable Gumbel-softmax operations select discrete entries during training.

  • Model architecture: The architecture combines a convolutional feature encoder f, a Transformer g producing contextual representations, and a quantization module that generates discrete targets.The encoder outputs z_1, . . . , z_T, the Transformer outputs c_1, . . . , c_T, and quantization maps encoder outputs to q_t.
  • Feature encoder: The feature encoder uses temporal convolutions, layer normalization, and GELU activations on normalized raw waveform input.The waveform is normalized to zero mean and unit variance, and encoder stride determines the Transformer’s number of time-steps T.
  • Contextualized representations with Transformers: The context network uses a Transformer with convolutional relative positional embeddings instead of fixed absolute positional embeddings.The convolution output is passed through GELU, added to the inputs, and followed by layer normalization.
  • Quantization module: Product quantization selects entries from G codebooks, concatenates them, and applies a linear transformation to produce each quantized representation q.Each codebook contains V entries e ∈ R^(V × d/G), and the concatenated vector is transformed from R^d to R^f.
  • Quantization module: Gumbel-softmax operations make codebook selection differentiable, using a straight-through estimator with G hard operations.The encoder output is mapped to G × V logits; forward selection uses argmax, while backward propagation uses the true Gumbel-softmax gradient.

3 Training

Training masks latent speech representations and pre-trains the model with a contrastive objective over quantized representations and distractors. The resulting model is fine-tuned for speech recognition using a linear projection and CTC loss.

  • Latent masking: Pre-training masks spans of latent encoder outputs before the context network, replacing them with a shared trained feature vector while leaving quantization inputs unmasked.Masking samples starting indices and masks subsequent consecutive time steps.
  • Contrastive objective: The contrastive task identifies the true quantized representation for each masked time step among K + 1 candidates containing K distractors.Distractors are uniformly sampled from other masked time steps in the same utterance.
  • Codebook diversity: A codebook diversity loss encourages equal use of the V entries in each of G codebooks by maximizing entropy over averaged codebook distributions.The diversity loss supplements the contrastive objective during pre-training.
  • Fine-tuning: Fine-tuning adds a randomly initialized linear projection from the context network to C vocabulary classes and optimizes a CTC loss.For Librispeech, the target vocabulary contains 29 character tokens plus a word boundary token.

4 Experimental Setup

The experiments pre-train on large unlabeled speech corpora and fine-tune across multiple labeled-data regimes, including Librispeech, Libri-light, and TIMIT. They evaluate BASE and LARGE Transformer configurations under specified masking, optimization, and decoding protocols.

  • Data: Pre-training uses 960 hours of unlabeled Librispeech audio or 53.2k hours from LibriVox, followed by fine-tuning on labeled Librispeech and Libri-light subsets.The labeled settings include 960 hours, 100 hours, and Libri-light limited-resource subsets.
  • Data: Phoneme recognition is evaluated on TIMIT’s five hours of recordings using its standard train, development, and test split with labels collapsed to 39 classes.The models predict phonemes for TIMIT and characters for Librispeech and Libri-light.
  • Pre-training: Masking samples p = 0.065 of time-steps as starts and masks M = 10 subsequent steps, covering approximately 49% of time-steps with 299ms mean spans.The implementation uses fairseq.
  • Model configurations: BASE uses 12 Transformer blocks with dimension 768, while LARGE uses 24 blocks with dimension 1,024 and trains on 64 and 128 V100 GPUs, respectively.BASE and LARGE differ in Transformer depth, dimensions, attention heads, batch construction, and training duration.
  • Optimization: Optimization uses Adam with warmup and linear decay; BASE trains for 400k updates, LARGE for 250k, and LARGE on LV-60k for 600k updates.The quantization module uses G = 2 and V = 320, yielding a theoretical maximum of 102.4k codewords.
  • Fine-tuning and decoding: Fine-tuning adds a randomly initialized output layer, uses tri-state learning-rate schedules, and decodes with 4-gram or Transformer language models tuned by Bayesian optimization.The 4-gram and Transformer language models are trained on the Librispeech language-model corpus.

5 Results

wav2vec 2.0 improves low-resource and full-resource speech recognition, achieving strong Librispeech and TIMIT results with limited labeled data. Performance benefits from larger models, more unlabeled data, and continuous inputs with quantized targets.

  • Low-resource speech recognition: 5.2/8.6 WER is achieved on Librispeech clean/other with only 10 minutes of labeled data and LV-60k pre-training.The 10-minute set contains 48 recordings averaging 12.5 seconds.
  • Low-resource speech recognition: 2.3/5.0 WER on the 100-hour Librispeech test-clean/other sets improves over iterative self-training’s 4.2/8.6 WER by 45%/42%.The proposed approach uses pre-training followed by fine-tuning rather than repeated labeling, filtering, and retraining.
  • Low-resource speech recognition: 3.2/6.1 WER with 10 labeled hours and 3.9/7.6 with one labeled hour outperform iterative self-training by 24%/29% and 7%/12%, respectively.The one-hour result uses two orders of magnitude less labeled data.
  • Phoneme recognition: 23%/29% relative PER reductions on TIMIT dev/test establish a new state of the art in phoneme recognition.The models are fine-tuned on labeled TIMIT training data without a language model.
  • Ablation study: Continuous inputs with quantized targets perform best, while quantizing both inputs and targets performs least well.Continuous latent representations retain more information, whereas quantized targets make training more robust.

6 Conclusion

wav2vec 2.0 uses latent-space masking and contrastive learning over quantized speech representations for self-supervised speech pre-training. With limited labeled data, it achieves strong Librispeech results, including 4.8/8.2 WER using only 10 minutes of labeled training data.

  • Framework: wav2vec 2.0 masks latent representations of raw waveforms and solves a contrastive task over quantized speech representations.The quantized representations are learned jointly within the framework.
  • Limited-label results: 4.8/8.2 WER is achieved on Librispeech test-clean/other using only 10 minutes of labeled training data.This corresponds to 48 recordings averaging 12.5 seconds.
  • Benchmark results: New state-of-the-art results are achieved on the full Librispeech benchmark for noisy speech.The conclusion characterizes these results as a new state of the art for noisy speech.
  • Benchmark results: 100 times less labeled data is used than the previous best result on the clean 100 hour Librispeech setup while wav2vec 2.0 achieves better performance.The approach also remains effective when large amounts of labeled data are available.
  • Future directions: Performance gains are expected from switching to a seq2seq architecture and a word piece vocabulary.These are proposed directions for improving the approach.

Broader Impact

The paper argues that limited annotated data could make speech recognition feasible for many of the world’s languages and dialects, where existing systems often require extensive labeling.

  • Access to Speech Recognition: Around 7,000 languages and many more dialects lack speech recognition technology.The passage attributes this gap to the difficulty of collecting labeled data for most languages.
  • Access to Speech Recognition: Current speech recognition systems require hundreds or thousands of hours of labeled data.That requirement is difficult to meet for most languages.
  • Access to Speech Recognition: The authors show that speech recognition models can achieve very good accuracy with very small amounts of annotated data.They hope this will broaden access to speech recognition across more languages and dialects.

Appendices … C Full results for Libri-light and Librispeech

The appendices specify the masking and fine-tuning procedures, then identify the tables reporting full WER results for Libri-light and Librispeech. The masking setup produces substantial, overlapping spans and masks channels during fine-tuning.

  • Appendices: The appendix describes masking hyperparameters and expands sampled starting time steps into spans that may overlap.Each latent representation is a candidate start with probability p, and M controls span length.
  • A Masking distribution: 49% of time-steps are masked in a 15-second sample, with an average span of 14.7 time-steps, or 299ms.The median span is 10 time-steps and the maximum is about 100 time-steps.
  • A Masking distribution: The masking-distribution appendix reports ablations of M, p, and alternative masking strategies in Table 5.It notes that reducing M increases self-supervised prediction accuracy, although the passage is truncated after this statement.
  • B Fine-tuning Setup: Fine-tuning randomly masks ten-time-step spans in feature-encoder outputs, allowing overlapping spans and reusing the pre-training mask embedding.It also masks channels by expanding selected starting indices across 64 subsequent channels.
  • B Fine-tuning Setup: Tables 6–8 document fine-tuning hyperparameters and decoding parameters for Librispeech and LibriVox pre-trained models across labeled-data setups.Table 6 covers fine-tuning settings, while Tables 7 and 8 cover decoding settings for Librispeech and LibriVox pre-training, respectively.
  • C Full results for Libri-light and Librispeech: Table 9 reports WER on Librispeech development and test sets for Libri-light low-resource labeled-data setups.The table corresponds to the low-resource results summarized in Table 1.
  • C Full results for Libri-light and Librispeech: Table 10 reports Librispeech WER when all 960 hours of Librispeech are used as labeled data.The table corresponds to the full-supervision results summarized in Table 2.

D Analysis of Discrete Latent Speech Representations

The analysis tests whether wav2vec 2.0’s discrete latent speech representations encode phonetic information by examining their co-occurrence with annotated phonemes on TIMIT. Many latents specialize in particular phonetic sounds, while silence is distributed across multiple latents.

  • Phonetic analysis: The study computes discrete latents for TIMIT training utterances using LARGE pre-trained on LV-60k, without fine-tuning, and measures their co-occurrence with annotated phonemes.Ties are resolved using the phoneme most represented in each latent’s receptive field.
  • Phonetic analysis: Many discrete latents specialize in specific phonetic sounds, according to P(phoneme|q_t) measured on TIMIT training data.The visualization uses collapsed 39 phoneme classes across the discrete latents.
  • Phonetic analysis: 22% of human-annotated speech data is silence (bcl), so it is modeled by many different discrete latents.This distribution contrasts with the specialization observed for many other phonetic sounds.

E Speech Recognition Error Analysis

The error analysis shows that limited-label models mainly make spelling and pronunciation-like errors, while increasing labeled data reduces these errors and shifts failures toward rarer words, names, and segmentation. A Transformer language model substantially improves the ten-minute model’s WER by selecting more likely pronunciations during decoding.

  • Ten-minute models: WER 38.3 on dev-clean falls to 5.0 for the LARGE LV-60k model when a Transformer language model is added in the ten-minute setup.The language model improves search by selecting more likely pronunciations.
  • Ten-minute models: Most ten-minute errors involve silent-character omissions, phonetic spellings, or omitted repeated letters, such as could → coud and still → stil.These errors show that the model recognizes basic speech units but has difficulty grounding them in exact spelling.
  • Scaling labeled data: At one and ten hours of labeled data, errors shift from basic spelling mistakes toward phonetic spellings of less common words and frequent article errors.Examples include heaven and food at one hour, and a and the at ten hours.
  • Scaling labeled data: At 100 hours, person names and incorrect word spacing dominate frequent errors, while at 960 hours WER falls to 2% and remaining errors are mostly articles, splits, rare words, or names.Examples include phoebe → phebe, anyone → any one, awhile → a while, deucalion, and gryce.
  • Pre-training comparison: The from-scratch 960-hour model has a similar WER and error pattern to the 100-hour pre-trained model.This comparison concerns the models’ overall word error rate and the types of errors they make.

F Ablations

Ablations show that the baseline design is generally robust, while codebook training signals, diversity regularization, and Gumbel noise are important. Alternative encoder, sampling, positional, receptive-field, and local-prediction choices generally provide little or no benefit, and some hurt performance.

  • Architecture: Adding an MLP to continuous targets or using separate input and target encoders did not yield meaningful improvements.These variants were tested against the baseline architecture.
  • Quantization and context: Too-low diversity penalty reduced codebook usage and performance, whereas too-high weighting caused slight instability.Increasing the receptive field from 25ms to 30ms and doubling relative positional embeddings to 256 had little or no benefit.
  • Quantization and context: Stopping quantizer gradients showed that the encoder requires training signal from the quantizer.This ablation directly tests gradient flow between quantization and the encoder.
  • Negative sampling: Increasing negatives to K = 200 did not improve performance, while sampling across utterances hurt because those candidates were easy to distinguish.Sampling from any masked or unmasked time step also failed to help and increased computational cost.
  • Quantization: Gumbel noise was important, whereas increasing the number of codebooks did not improve performance.Both choices were evaluated as pre-training quantization ablations.
  • Prediction targets: Predicting only time steps next to each masked span was investigated to control pre-training difficulty, with prediction of only one step performing poorly.The local contrastive loss uses masked steps adjacent to the leftmost or rightmost unmasked step.
Loading 2006.11477v3…