Source-linked AI summary
W2v-BERT: Combining Contrastive Learning and Masked Language Modeling for Self-Supervised Speech Pre-Training
Yu-An Chung, Yu Zhang, Wei Han, Chung-Cheng Chiu, James Qin, Ruoming Pang, Yonghui Wu
TL;DR
Self-supervised speech pre-training must exploit large-scale unlabeled audio while learning representations useful for ASR. w2v-BERT jointly combines contrastive discretization and masked language modeling, and it matches or outperforms state-of-the-art systems across LibriSpeech and a more challenging internal dataset.
Problem
The paper seeks to improve unsupervised pre-training for semi-supervised ASR using large-scale unannotated speech.
Method
w2v-BERT jointly optimizes contrastive learning to discretize speech and masked prediction to learn contextualized speech representations.
Results
w2v-BERT either outperforms or matches state-of-the-art systems on LibriSpeech and transfers gains to a more challenging internal dataset.
Takeaways & Limitations
The results support combining contrastive learning and masked prediction for self-supervised speech representation learning.
Takeaways & Limitations
Voice-search traffic is noisier, contains more silences, and has shorter audio than read speech, making it more challenging for pre-training.
Abstract
from arXiv · showhide
Motivated by the success of masked language modeling~(MLM) in pre-training natural language processing models, we propose w2v-BERT that explores MLM for self-supervised speech representation learning. w2v-BERT is a framework that combines contrastive learning and MLM, where the former trains the model to discretize input continuous speech signals into a finite set of discriminative speech tokens, and the latter trains the model to learn contextualized speech representations via solving a masked prediction task consuming the discretized tokens. In contrast to existing MLM-based speech pre-training frameworks such as HuBERT, which relies on an iterative re-clustering and re-training process, or vq-wav2vec, which concatenates two separately trained modules, w2v-BERT can be optimized in an end-to-end fashion by solving the two self-supervised tasks~(the contrastive task and MLM) simultaneously. Our experiments show that w2v-BERT achieves competitive results compared to current state-of-the-art pre-trained models on the LibriSpeech benchmarks when using the Libri-Light~60k corpus as the unsupervised data. In particular, when compared to published models such as conformer-based wav2vec~2.0 and HuBERT, our model shows~5\% to~10\% relative WER reduction on the test-clean and test-other subsets. When applied to the Google's Voice Search traffic dataset, w2v-BERT outperforms our internal conformer-based wav2vec~2.0 by more than~30\% relatively.
1. INTRODUCTION
w2v-BERT addresses unsupervised pre-training for semi-supervised ASR by jointly combining contrastive learning and masked prediction. It produces discretized speech units and contextualized representations in an end-to-end framework, with strong LibriSpeech and voice-search results.
- Motivation: Unsupervised pre-training uses unlabeled speech proxy tasks to initialize models before supervised ASR fine-tuning.The paper focuses on improving this aspect of semi-supervised ASR.
- Proposed framework: w2v-BERT combines wav2vec 2.0’s contrastive task with BERT-style masked prediction for self-supervised speech representation learning.Contrastive learning obtains discriminative discretized speech units, which serve as targets for masked prediction.
- Proposed framework: The contrastive and masked prediction objectives are optimized simultaneously in an end-to-end training framework.This avoids requiring the masked prediction task to wait for a separately completed contrastive-training stage.
- Results: w2v-BERT achieves state-of-the-art performance on the LibriSpeech task.The experiments use the Libri-Light unlab-60k corpus for pre-training and evaluate on LibriSpeech benchmarks.
- Results: w2v-BERT greatly improves a real-world voice-search recognition task over conformer-based wav2vec 2.0.The paper also analyzes contrastive learning’s necessity and the usefulness of mask prediction for alleviating easy negative samples.
- Architecture: The framework consists of a feature encoder, a contrastive module, and a masked language modeling module built from conformer blocks.N and M denote the numbers of conformer blocks in the contrastive and MLM modules, respectively.
2. RELATED WORK
w2v-BERT shares with related methods the strategy of discretizing speech before masked language modeling, but differs in how its modules are trained and constructed. Its end-to-end design and conformer architecture distinguish it from several prior frameworks.
- Shared approach: w2v-BERT, HuBERT, vq-wav2vec, and DiscreteBERT transform continuous speech into discretized units for masked language modeling.These methods share the high-level goal of learning contextualized speech representations from discrete speech units.
- Training design: Unlike vq-wav2vec and DiscreteBERT, w2v-BERT trains its speech discretizer and contextualized representation module end-to-end.The other methods use a two-stage process with a pre-obtained, frozen discretizing module.
- Architecture: w2v-BERT uses conformer layers that combine convolutional and transformer components to model local and global dependencies in audio sequences.The paper contrasts this construction with related methods that use transformer layers as building blocks.
- Empirical distinction: w2v-BERT outperforms w2v-Conformer, supporting the effectiveness of the pre-training framework beyond the choice of conformer building blocks.Both methods use conformer layers, so the comparison is presented as evidence that architecture alone does not explain the result.
- Comparison with wav2vec 2.0: Unlike w2v-BERT, wav2vec 2.0 employs contrastive learning without combining it with masked prediction.The paper states that wav2vec 2.0’s resulting ASR performance lags behind that of combining contrastive learning and masked prediction.
3. METHOD
w2v-BERT combines feature encoding, contrastive discretization, and masked prediction in an end-to-end self-supervised architecture. It jointly trains the contrastive and masked prediction objectives using unlabeled speech.
- Model architecture: The architecture contains a feature encoder, a contrastive module that produces discretized speech tokens, and a masked prediction module for contextualized representations.The feature encoder extracts latent speech representations from raw acoustic inputs; the subsequent modules process them for tokenization and contextual learning.
- Contrastive learning: The contrastive module masks feature-encoder outputs to produce context vectors while quantizing unmasked outputs into vectors and token IDs.The context vectors and quantized vectors support the wav2vec 2.0 contrastive task, while token IDs become targets for masked prediction.
- Masked prediction: The masked prediction module transforms contrastive context vectors into contextualized speech representations and predicts token IDs at masked positions.A softmax layer is applied to the final conformer block to predict the corresponding token ID.
- Contrastive learning: The contrastive loss combines the wav2vec 2.0 loss Lw with codebook diversity loss Ld, weighted by α.The diversity term encourages uniform usage of codes, with α = 0.1 in the experiments.
- Joint training: w2v-BERT minimizes a final loss that jointly trains the contrastive and masked prediction tasks, using β = 1 and γ = 1 in the experiments.During pre-training, only unlabeled speech data is used.
- Fine-tuning: During fine-tuning, the pretrained model is used in a sequence-transducer ASR network with a projection block and LSTM decoder.The projection block contains a linear layer with Swish activation and batch normalization.
4. EXPERIMENTAL SETUP
The experiments pre-train w2v-BERT on Libri-Light and evaluate it on LibriSpeech using supervised fine-tuning, while also applying practical ASR improvements. Two model scales and multiple training settings are compared.
- Data and evaluation: w2v-BERT is pretrained on about 60,000 hours of unannotated Libri-Light speech and mainly fine-tuned with the LibriSpeech 960hr supervised subset.The 100hr LibriSpeech subset is used for ablation studies, with WER reported on four evaluation subsets.
- Data and evaluation: The evaluation reports WER on dev-clean, dev-other, test-clean, and test-other using 80-dimensional log-mel filter bank acoustic inputs.Transcript tokenization uses a 1024-token WordPiece model.
- Pre-training details: Masking samples feature-encoder starting positions with probability 0.065 and masks the subsequent 10 time steps, allowing overlap.The masking setup follows the referenced wav2vec 2.0 and w2v-Conformer configurations.
- Pre-training details: Two pretrained variants contain about 0.6 billion and 1 billion parameters, named w2v-BERT XL and w2v-BERT XXL.XL uses 12 conformer blocks in each major module, while XXL expands its masked prediction module to 30 blocks.
- Fine-tuning and comparison: LibriSpeech comparisons include training from scratch, self-training only, pre-training only, and combined pre-training plus self-training settings.Table 2 reports WERs and includes model sizes, language-model settings, and selected literature results.
- Fine-tuning and comparison: Fine-tuning additionally uses SpecAugment, Noisy Student Training, and language-model fusion according to the setup in the referenced work.These techniques are applied as practical performance-improvement methods alongside self-supervised pre-training.
5. RESULTS AND DISCUSSION
w2v-BERT combines contrastive learning and masked prediction in an end-to-end framework, with experiments showing strong LibriSpeech results and substantial gains over contrastive-only baselines. Analyses find that contrastive learning is necessary for preventing code collapse and that model capacity must be balanced between contrastive and masked-prediction modules.
- Main results: w2v-BERT XL achieves WERs of 1.5/2.9 on LibriSpeech test/test-other without self-training or LM, outperforming or matching other models using LM.w2v-BERT XXL further increases the gap on the more challenging dev-other and test-other subsets.
- Main results: 28%, 42%, 32%, and 38% relative WER reductions are achieved by w2v-BERT-XXL versus wav2vec 2.0 across four evaluation subsets without LM.With LM, the corresponding reductions are 13%, 13%, 17%, and 18%.
- Main results: w2v-BERT XL outperforms w2v-Conformer-XXL despite having fewer parameters, especially on dev-other and test-other, while w2v-BERT-XXL performs better at comparable size.The comparison keeps model size and fine-tuning pipeline aligned, isolating the pre-training method.
- Analysis and discussion: Contrastive learning supplies a discriminative codebook that enables masked prediction and prevents the trivial solutions that arise when the contrastive module is removed.Without contrastive learning, masked prediction loss approaches zero, accuracy reaches 100%, and diversity loss approaches one, indicating code collapse.
- Analysis and discussion: WERs mostly decrease as the contrastive module grows from C2 to C8, but deeper contrastive modules eventually stop helping when the masked-prediction module becomes too small.Increasing the masked-prediction module while holding the contrastive module fixed can still improve performance, as shown by w2v-BERT XL versus XXL.
- Analysis and discussion: w2v-BERT consistently outperforms wav2vec 2.0 across contrastive-module configurations and matches or exceeds w2v-Conformer XL when the contrastive module has sufficient capacity.On voice-search traffic, w2v-BERT XL improves an identically configured tuned contrastive baseline by relative 30%.
6. CONCLUSION AND FUTURE WORK
w2v-BERT combines contrastive discretization and masked language modeling in jointly optimized modules for self-supervised speech representation learning. Pre-training on 60k hours of Libri-Light speech matches or outperforms state-of-the-art systems, while hyperparameter optimization and low-resource evaluation remain future work.
- w2v-BERT jointly optimizes contrastive speech discretization and masked language modeling in two modules.The contrastive module discretizes continuous speech, while the masked prediction module performs masked language modeling on the discretized speech.
- 60k hours of Libri-Light pre-training lets w2v-BERT match or outperform state-of-the-art systems including w2v-Conformer, HuBERT, and wav2vec 2.0.The gains also transfer to a more challenging internal dataset.
- Future work: The experiments reuse all hyperparameter setups directly from [21] without changes.The authors identify searching for the best w2v-BERT training configuration as future work.
- Future work: Evaluating w2v-BERT on Libri-Light's 10min, 1hr, and 10hr low-resource benchmarks is planned future work.