PaperlayerView exact revision on arXiv

Source-linked AI summary

WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing

Sanyuan Chen, Chengyi Wang, Zhengyang Chen, Yu Wu, Shujie Liu, Zhuo Chen, Jinyu Li, Naoyuki Kanda, Takuya Yoshioka, Xiong Xiao, Jian Wu, Long Zhou, Shuo Ren, Yanmin Qian, Yao Qian, Jian Wu, Michael Zeng, Xiangzhan Yu, Furu Wei

arXiv:2110.13900v5cs.CLcs.SDeess.AS

TL;DR

Existing self-supervised speech models have limited support for diverse, especially non-ASR and multi-speaker, tasks. WavLM combines masked speech prediction with denoising and large-scale pre-training, achieving state-of-the-art results across SUPERB and representative speech-processing benchmarks.

  • Problem

    Existing self-supervised speech models provide limited support for diverse speech tasks, particularly multi-speaker tasks, motivating a general full-stack pre-trained model.

  • Method

    WavLM pre-trains universal speech representations by combining masked speech prediction with denoising, gated relative position bias, and 94k hours of unlabeled audio.

  • Results

    WavLM Large outperforms HuBERT Large on 14 SUPERB subtasks and improves the overall evaluation by an absolute 2.4 points.

  • Takeaways & Limitations

    WavLM is effective across ASR, speaker verification, speech separation, and speaker diarization, supporting its use as a general speech-processing backbone.

Abstract

from arXiv · show

Self-supervised learning (SSL) achieves great success in speech recognition, while limited exploration has been attempted for other speech processing tasks. As speech signal contains multi-faceted information including speaker identity, paralinguistics, spoken content, etc., learning universal representations for all speech tasks is challenging. To tackle the problem, we propose a new pre-trained model, WavLM, to solve full-stack downstream speech tasks. WavLM jointly learns masked speech prediction and denoising in pre-training. By this means, WavLM does not only keep the speech content modeling capability by the masked speech prediction, but also improves the potential to non-ASR tasks by the speech denoising. In addition, WavLM employs gated relative position bias for the Transformer structure to better capture the sequence ordering of input speech. We also scale up the training dataset from 60k hours to 94k hours. WavLM Large achieves state-of-the-art performance on the SUPERB benchmark, and brings significant improvements for various speech processing tasks on their representative benchmarks. The code and pre-trained models are available at https://aka.ms/wavlm.

I. INTRODUCTION · II. RELATED WORK

WavLM targets universal representations for full-stack speech processing by combining masked speech prediction with denoising, while addressing limitations of prior speech SSL models. It introduces gated relative position bias and large-scale training, achieving strong results across ASR, speaker, and multi-speaker tasks.

  • I. INTRODUCTION: Speech SSL has succeeded especially on phoneme classification and ASR, but other speech tasks remain insufficiently explored.The paper motivates broader SSL because speech contains information relevant to diverse tasks.
  • I. INTRODUCTION: A general pre-trained model could improve data-scarce non-ASR tasks, reduce labeling effort, and avoid building separate models for each downstream task.The motivation is large-scale pre-training on unlabeled speech data.
  • I. INTRODUCTION: Existing models remain unsatisfactory for multi-speaker tasks, with HuBERT-based speech separation yielding only marginal improvement over training from scratch.The paper attributes this limitation to pre-training methods that insufficiently encode information needed by multi-speaker processing.
  • I. INTRODUCTION: WavLM combines masked speech prediction and denoising on simulated noisy or overlapped speech to learn both ASR information and non-ASR capabilities.Pseudo-label prediction on overlapped speech supports diarization and separation, while noisy speech modeling captures speaker identity and speech enhancement.
  • I. INTRODUCTION: Gated relative position bias adapts positional information to current speech content while improving ASR performance with nearly unchanged parameter count and training speed.The method is presented as an optimization of the HuBERT and wav2vec 2.0 model structures.
  • I. INTRODUCTION: WavLM Large outperforms HuBERT Large on 14 subtasks and improves the overall SUPERB evaluation by an absolute 2.4 points.WavLM achieves state-of-the-art performance on SUPERB, while WavLM Base+ is better than HuBERT Large despite being 3 times smaller.
  • I. INTRODUCTION: 0.383%, 0.480% and 0.986% EER are achieved by WavLM Large on the three official VoxCeleb1 trial lists, exceeding ECAPA-TDNN by a large margin.Speaker verification evaluates extraction of speaker-related features.
  • I. INTRODUCTION: 27.7% relative WER reduction is achieved over Conformer on LibriCSS, while WavLM also reports SOTA on CALLHOME diarization with a 12.6% diarization error rate reduction over EEND-EDA.These results cover speech separation and speaker diarization, respectively.

III. BACKGROUND: HUBERT · IV. WAVLM · A. Model Structure

HuBERT uses masked prediction over clustered acoustic targets, whereas WavLM extends this paradigm with masked speech denoising for broader speech processing. WavLM combines a convolutional feature encoder with a Transformer using content-adaptive gated relative position bias.

  • III. BACKGROUND: HUBERT: HuBERT uses a Transformer encoder to predict discrete clustered acoustic targets from masked acoustic features.The prediction loss is applied only over masked regions.
  • III. BACKGROUND: HUBERT: HuBERT iteratively re-clusters MFCC features and later latent representations to generate successive training-target generations.The first iteration clusters MFCC features, while the second clusters representations from the trained first-iteration model.
  • IV. WAVLM: WavLM jointly masks noisy or overlapped inputs and predicts pseudo-labels of the original speech on masked regions.This masked speech denoising framework is designed to retain speech-content modeling while representing information useful for non-ASR tasks.
  • IV. WAVLM: WavLM extends pre-training to 94k public English data and targets speaker identification, separation, and diarization alongside ASR.The denoising objective implicitly models information needed for these non-ASR tasks.
  • A. Model Structure: The model uses a convolutional feature encoder followed by a Transformer encoder, with seven temporal-convolution blocks producing approximately 25ms representations every 20ms.The convolutions use 512 channels, strides (5,2,2,2,2,2,2), and kernel widths (10,3,3,3,3,2,2).
  • A. Model Structure: WavLM adds gated relative position bias based on key-query offsets in Transformer self-attention.The relative position embedding uses n = 320 shared embeddings, with logarithmic ranges up to maximum offset m = 800.
  • A. Model Structure: Unlike convolutional relative position embeddings in wav2vec 2.0 and HuBERT, WavLM gates adaptively adjust positional bias according to current speech content.The same frame-distance offset can play different roles when frames differ in silence versus speech content.

B. Masked Speech Denoising and Prediction

The framework masks manually simulated noisy or overlapped speech and predicts pseudo-labels for the original speech, aiming to improve robustness in complex acoustic environments while preserving speaker identity.

  • Framework: Masked speech denoising predicts pseudo-labels of the original speech on masked regions from noisy or overlapped inputs.The framework manually simulates the corrupted speech inputs for self-supervised pre-training.
  • Input simulation: Training mixtures combine utterances with randomly selected noise audio or secondary utterances at random regions.Noise and secondary utterances are randomly cropped and scaled by a random source energy ratio; overlap is kept below 50%.

1) Noisy/Overlapped Speech Simulation:

Algorithm 1 simulates noisy or overlapped speech by selecting utterances from a batch, mixing each with either another utterance or DNS noise, and sampling mixing conditions. Uniformly sampled positions and a mixing portion below 50% preserve a distinguishable primary speaker.

  • Noisy/Overlapped Speech Simulation: Given a batch of B speech utterances of length L and M DNS noises, the method randomly selects S utterances using Bernoulli sampling with probability p.The utterances and noises are denoted U = {ui}B and N = {ni}M.
  • Noisy/Overlapped Speech Simulation: For each selected primary utterance, a uniform random value determines whether to mix DNS noise or a secondary batch utterance.The decision uses v sampled from U(0, 1) and compares it with the mixing noise probability pn.
  • Noisy/Overlapped Speech Simulation: The mixing energy ratio is sampled from U(−5, 5) for overlapped speech and U(−5, 20) for noise mixing.These ranges follow the typical training utterance simulation process for speech separation.
  • Noisy/Overlapped Speech Simulation: Primary and secondary start positions are uniformly sampled from {1, · · · , L −l}, while each mixing portion remains below 50%.Constraining the mixing portion below 50% ensures the primary utterance remains longer than the secondary utterance, avoiding an indistinguishable main speaker.

2) Mask Prediction Loss: · C. Pre-Training Data · D. Stabilization of Training

The model uses HuBERT-style masked prediction with pseudo-labels generated from the original utterance, while extending pre-training data beyond audiobooks and stabilizing fp16 training against attention-score overflow. These choices support non-ASR information modeling, broader robustness, and numerically safer optimization.

  • 2) Mask Prediction Loss:: Pseudo-labels z are generated by feeding utterance u to the previous-iteration network and clustering MFCC or latent representations with k-means.The simulated utterance u′ is processed by the current network for masked prediction.
  • 2) Mask Prediction Loss:: The masked prediction framework implicitly models non-ASR information during pre-training, benefiting diverse non-ASR tasks.
  • C. Pre-Training Data: Large-scale unsupervised data from diverse domains is used because audiobook-only LibriSpeech or LibriLight pre-training limits generalization to varied acoustic conditions.Real captured speech commonly contains background acoustics unlike audiobook recordings.
  • C. Pre-Training Data: 10k hours of GigaSpeech data expand training coverage across audiobooks, podcasts, YouTube, read and spontaneous speech, and varied topics.The full GigaSpeech collection contains 40k hours, but 30k hours were not well processed.
  • D. Stabilization of Training: fp16 or mixed-precision pre-training can become unstable for large models because attention-score computation may overflow, producing NaN losses.
  • D. Stabilization of Training: The stabilization trick exploits softmax invariance to translating every coordinate by the same constant before computing attention probabilities.This reformulation preserves the softmax result while controlling attention-score magnitudes.
  • D. Stabilization of Training: The scale hyperparameter c is set to 32, ensuring the maximum attention-score term remains below 2^16 and resolving the overflow issue.

V. EXPERIMENT … 2) Evaluation result:

The experiments evaluate WavLM as a universal speech representation on SUPERB under standardized frozen-model settings. WavLM Base+ and WavLM Large outperform strong SSL baselines, while larger and more diverse pre-training data and model scaling improve broad downstream performance.

  • A. Pre-Training Setup: WavLM Base and Base+ use 12 Transformer layers, 768-dimensional hidden states, 8 attention heads, and 94.70M parameters, while WavLM Large uses 24 layers, 1024-dimensional states, 12 heads, and 316.62M parameters.Relative position embeddings are shared across all layers to avoid significantly increasing parameter count.
  • 1) Setup:: SUPERB evaluates pretrained speech representations across fifteen tasks spanning speaker, content, paralinguistic, and other speech-processing capabilities.The benchmark includes tasks such as speaker identification, speaker verification, diarization, phoneme recognition, and automatic speech recognition.
  • 1) Setup:: The evaluation follows SUPERB settings, using the same downstream models while freezing pretrained models and consuming weighted sums of hidden states from every layer.Freezing limits the fine-tuning hyperparameter search space.
  • 2) Evaluation result:: WavLM Base+ outperforms HuBERT large and wav2vec 2.0 large on the overall SUPERB score, demonstrating strong universal representation learning.The overall score is computed using task-specific score transformations described in the evaluation setup.
  • 2) Evaluation result:: WavLM Base performs better than wav2vec 2.0 Base and HuBERT Base on all downstream tasks under a fair comparison using equal pre-training data and parameter counts.The results support the effectiveness of WavLM’s structure and masked speech denoising modeling.
  • 2) Evaluation result:: WavLM Base+ consistently improves WavLM Base and surpasses wav2vec 2.0 Large and HuBERT Large overall, showing the contribution of larger and more diverse pre-training data.The combined dataset especially improves ASV, OOD-ASR, IC, SF, and ER test sets that are not extracted from audiobooks.
  • 2) Evaluation result:: 38% word error rate reduction on ASR results from scaling to WavLM Large, while SID improves by 6.07% absolutely; WavLM Large also beats HuBERT Large across 14 downstream tasks.The larger model benefits most tasks, especially ASR, and also improves speaker-related performance.

3) Analysis: … 3) Setup:

The analysis examines layer contributions in SUPERB, while the speaker-verification setup evaluates pre-trained representations with ECAPA-TDNN under specified data, training, and scoring procedures.

  • 3) Analysis:: Layer weights are normalized by corresponding hidden-state values before task-specific layers consume their weighted sum.This removes bias toward layers with larger hidden-state values.
  • 3) Analysis:: Bottom layers contribute more to speaker-related tasks, whereas top layers are more important for recognition and language-understanding tasks in Base models.The pattern is similar between WavLM and HuBERT.
  • 3) Analysis:: Four typical speech tasks are additionally used to probe model limits because frozen SUPERB fine-tuning cannot fully expose pre-trained-model capacity.Their training-data amounts are not on the same scale.
  • C. Speaker Verification: Speaker verification determines whether a reference clip comes from the same speaker as a given audio clip.Training data consist of audio and speaker-ID pairs, D = {x_i, y_i}.
  • 2) Datasets:: VoxCeleb1 and VoxCeleb2 provide the speaker-verification experiments, with online MUSAN, DNS-noise, and RIR augmentation applied at probability 0.6.Evaluation uses all official Vox1-O, Vox1-E, and Vox1-H trial lists without VAD processing.
  • 3) Setup:: ECAPA-TDNN (small) compares handcrafted features with pre-training features for speaker verification.It uses a frame encoder, statistic pooling, and a fully connected layer to extract speaker embeddings.
  • 3) Setup:: Training uses 3s segments, AAM loss with margin 0.2, and an Inter-TopK penalty on 5 easily misclassified centers with penalty margin 0.1.Fbank systems train for 165 epochs; pre-trained systems are fixed for 20 epochs, then jointly fine-tuned for 5 more epochs.
  • 3) Setup:: Whole utterances produce speaker embeddings scored by cosine similarity, with adaptive s-norm using an imposter cohort of size 600.The cohort is estimated from the VoxCeleb2 development set by speaker-wise averaging extracted embeddings.

4) Results: … 3) Implementation Details:

The section reports strong speaker-verification performance for WavLM representations and describes the speaker-diarization formulation, datasets, and implementation used for evaluation. WavLM Base+ and Large outperform the SOTA ECAPA-TDNN system, while the diarization system predicts speakers and labels using segmented recordings and pre-trained representations.

  • 4) Results:: WavLM Base+ and Large representations outperform the SOTA ECAPA-TDNN system on speaker verification.All systems with pre-trained representations exceed the Fbank baseline on Vox1-O and Vox1-E, although HuBERT Base is slightly worse on Vox1-H.
  • 4) Results:: 0.461, 0.634, and 0.993 are the Vox1-O, Vox1-E, and Vox1-H results of the VoxSRC challenge 2021 winner system, respectively.After large margin fine-tuning and quality-aware score calibration, the best WavLM system exceeds that winner on all three trials.
  • 1) problem formulation:: Speaker diarization assigns one or more speaker-identity labels to each frame of a recording.Multiple labels indicate speaker overlap, and the system must predict both the recording’s speaker number and frame-level speaker labels.
  • 2) Datasets:: The experiments split data into large-scale simulated training data and real data for evaluation and adaptation.The simulation setup leverages speech from Switchboard, NIST Speaker Recognition Evaluation datasets, noises, and simulated room impulse responses.
  • 3) Implementation Details:: The downstream diarization model segments long recordings into short blocks assumed to contain at most SLocal speakers, with SLocal = 3.Mel-filterbank features feed a Transformer encoder that produces diarization results and SLocal speaker embeddings, optimized with diarization and speaker losses.
  • 3) Implementation Details:: 15s, 30s, and 30s are the block lengths for training, adaptation, and evaluation, respectively.Constrained AHC performs embedding clustering during evaluation.
  • 3) Implementation Details:: Pre-trained representations replace the handcrafted Fbank feature in the diarization system.The implementation follows the setup described for section V-C3, while evaluation uses constrained AHC for embedding clustering.

4) Results: … 3) Implementation details:

The paper evaluates WavLM on speaker diarization and speech separation, using mask-based separation with Conformer downstream models and pretrained representations. WavLM representations improve diarization over the implemented baseline and HuBERT, while separation combines duplicated pretrained features with STFT features for mask estimation.

  • 4) Results:: WavLM representations outperform handcrafted features in the evaluated CALLHOME speaker-diarization systems, with WavLM Base+ surpassing HuBERT large.The comparison reports that replacing handcrafted features improves all systems and that WavLM representations are more useful than HuBERT for diarization.
  • E. Speech Separation: Speech separation estimates individual source signals from mixtures that may contain entirely or partially overlapping speakers.The formulation represents source and mixture signals using STFTs and estimates source masks rather than directly predicting source STFTs.
  • 1) problem formulation:: The separation method estimates masks from the mixed-signal STFT and reconstructs each source STFT through elementwise mask multiplication.For each source, the reconstruction is Xs = Ms ⊙Y.
  • 2) Datasets:: The separation training set contains 219 hours of artificially reverberated and mixed WSJ1 utterances spanning four mixture types.Mixtures randomly select one or two speakers, apply image-method room impulse responses, and rescale signals before mixing.
  • 3) Implementation details:: The baseline is a Conformer separation model with 16 encoder layers, 4 attention heads, 256 attention dimensions, and 1024 FFN dimensions.A linear projection and sigmoid activation produce the predicted masks.
  • 3) Implementation details:: Fine-tuning uses WavLM as a feature extractor, concatenating its time-aligned representation with the STFT representation before downstream mask estimation.The pretrained representation is duplicated along time to match the STFT sequence length, then concatenated feature-wise.
  • 3) Implementation details:: Training uses AdamW with 1e-2 weight decay, a 1e-4 learning rate, 10,000 warm-up steps, and 260,000 total steps.The schedule uses linear decay after warm-up.
  • 3) Implementation details:: Evaluation uses an end-to-end Transformer ASR model whose reference WERs are 2.08% on LibriSpeech test-clean and 4.95% on test-other.The ASR model follows prior evaluation work and supplies the WER metric for separation assessment.

4) Results: … 1) Problem Formulation:

WavLM achieves strong speech-separation results on LibriCSS, outperforming baselines across overlap ratios, while layer-weight analysis indicates that downstream performance primarily draws on bottom Transformer layers. The speech-recognition formulation maps an input signal to its corresponding transcription.

  • 4) Results:: WavLM Base+ and Large achieve state-of-the-art single-channel utterance-wise separation results across all LibriCSS overlap ratios.Both models use frozen pre-trained parameters and outperform baseline results by a large margin.
  • 4) Results:: HuBERT Base representations produce performance comparable to the strengthened baseline across all overlap ratios.The authors attribute HuBERT’s limitation to limited optimization on speaker-overlapped speech and insufficient multi-speaker modeling during pre-training.
  • 4) Results:: WavLM Base+ reduces WER, particularly for audios with large overlap ratios, while freezing pre-trained parameters outperforms fine-tuning for evaluation.Fine-tuning improves training accuracy but worsens evaluation results, consistent with overfitting to artificially mixed training data.
  • 5) Weight Analysis:: Downstream speaker verification, diarization, and separation models receive weighted sums of representations from different pre-trained-model layers.The layer weights are analyzed for WavLM Base+ and WavLM Large.
  • 5) Weight Analysis:: The contribution to speaker verification, diarization, and separation mostly comes from the bottom layers.This observation is reported for the layer-weight analysis of WavLM Base+ and Large.
  • 1) Problem Formulation:: Speech recognition generates a transcription y = (y1, ..., yL) from an input speech signal x = (x1, ..., xT).T and L denote the lengths of the speech signal and transcription, respectively.

2) Datasets: … VI. CONCLUSION

The paper evaluates WavLM across LibriSpeech ASR settings and reports strong low-resource and full-data results, while concluding that its 94k-hour pre-training supports both ASR and non-ASR speech tasks. It also outlines implementation choices and future directions including scaling, compression, and joint text-speech learning.

  • 2) Datasets:: LibriSpeech experiments use 960 hours, train-clean-100, train-10h, and train-1h labeled-data partitions under the Libri-Light evaluation protocol.The splits contain 960, 100, 10, and 1 hours of labeled data, respectively.
  • 3) Implementation Details:: ASR fine-tuning adds a randomly initialized linear projection to the Transformer encoder and optimizes a CTC objective over 29 character tokens plus a word-boundary token.A modified SpecAugment procedure is also applied.
  • 3) Implementation Details:: Fine-tuning masks spans of ten time-steps and expands selected channel starts across 64 channels, setting selected channel spans to zero.Time-step spans use mask embeddings, may overlap, and reuse the pre-training masked-step embedding.
  • 3) Implementation Details:: The convolutional encoder remains fixed, the Transformer is frozen for the first 10k steps, and Adam uses a warmup, constant, then linear-decay learning-rate schedule.Base and Base+ use 8 GPUs, while Large uses 24 GPUs.
  • 3) Implementation Details:: Evaluation uses wav2letter++ beam search with language-model fusion, considering both a 4-gram and a Transformer language model.The language-model and word-insertion weights follow the specified fused-decoding formulation and prior evaluation hyperparameters.
  • 4) Results:: Without LM fusion, WavLM Base outperforms wav2vec 2.0 by a large margin across the 1-hour, 10-hour, and 100-hour fine-tuning splits.The low-resource comparison includes DeCoAR 2.0, DiscreteBERT, wav2vec 2.0, and HuBERT.
  • 4) Results:: On the full 960-hour LibriSpeech setting, pre-training methods outperform all supervised models, while WavLM is on par with the two best pre-training results.These findings are reported alongside the low-resource results in Table V and Table VI.
  • VI. CONCLUSION: WavLM uses 94k hours of audio and combines masked speech prediction with denoising to perform well on ASR and non-ASR tasks, updating state-of-the-art results across several benchmarks.The cited tasks include SUPERB, speaker verification, speech separation, and speaker diarization.

APPENDIX A HYPERPARAMTERS FOR PRE-TRAINING · SETTINGS OF DOWNSTREAM TASKS · HYPERPARAMTERS FOR FINE-TUNING

Appendix A documents WavLM’s pre-training configurations, downstream-task implementations, and fine-tuning hyperparameters. The settings follow prior work and SUPERB policies, with task-specific details and labeled-data setups summarized in Tables VII–X.

  • APPENDIX A HYPERPARAMTERS FOR PRE-TRAINING: Table VII lists the pre-training hyperparameters for WavLM Base, Base+, and Large, adapted from previous work.Training uses 32 V100 GPUs for Base and 64 V100 GPUs for Large.
  • APPENDIX A HYPERPARAMTERS FOR PRE-TRAINING: Table VII specifies batch-size computation in seconds and the GPU allocations used for Base and Large model training.The table caption states 32 V100 GPUs for Base and 64 V100 GPUs for Large.
  • SETTINGS OF DOWNSTREAM TASKS: All SUPERB universal-representation tasks use identical settings in accordance with SUPERB policies.These settings apply across the SUPERB task suite.
  • SETTINGS OF DOWNSTREAM TASKS: Speaker verification, speaker diarization, speech separation, and speech recognition use implementations shown in Table VIII, following previous work.For speaker diarization, CallHome is used for domain adaptation.
  • HYPERPARAMTERS FOR FINE-TUNING: Table IX gives learning-rate and batch-size hyperparameters for fine-tuning WavLM models on SUPERB downstream tasks.For speech translation, batch size denotes the number of tokens in each training batch.
  • HYPERPARAMTERS FOR FINE-TUNING: QbE uses dynamic time warping without fine-tuning, and all three WavLM models achieve their best results from the last-layer representations.Other downstream-task hyperparameters follow the official SUPERB implementation.
  • HYPERPARAMTERS FOR FINE-TUNING: Speech-recognition fine-tuning hyperparameters are summarized in Table X for different labeled-data setups.Table X specifically covers fine-tuning WavLM models for speech recognition.
Loading 2110.13900v5…