PaperlayerView exact revision on arXiv

Source-linked AI summary

Self-Supervised Speech Representation Learning: A Review

Abdelrahman Mohamed, Hung-yi Lee, Lasse Borgholt, Jakob D. Havtorn, Joakim Edin, Christian Igel, Katrin Kirchhoff, Shang-Wen Li, Karen Livescu, Lars Maaløe, Tara N. Sainath, Shinji Watanabe

arXiv:2205.10643v3cs.CLcs.SDeess.AS

TL;DR

Limited labeled data makes specialist supervised speech models difficult to extend across languages, dialects, and domains. This review surveys self-supervised speech representation methods, their historical and methodological connections, and benchmarks beyond recognition. It reports strong and broadly competitive performance across downstream tasks, while emphasizing the need for more comparable evaluation.

  • Problem

    Label-intensive supervised speech learning limits progress in languages and domains with scarce labeling, motivating reusable representations from unlabeled audio.

  • Method

    The review categorizes speech SSL methods as generative, contrastive, or predictive and surveys their benchmarks, downstream applications, and representation analyses.

  • Results

    SSL models are competitive across ASR, intent classification, speaker identification, automatic speaker verification, and query-by-example tasks, with wav2vec 2.0 and HuBERT achieving favorable WERs using 1 hour or 10 minutes of labels.

  • Takeaways & Limitations

    The reviewed evidence suggests SSL benefits generalize across tasks requiring content, speaker, and semantic information, supporting broader evaluation beyond ASR.

Abstract

from arXiv · show

Although supervised deep learning has revolutionized speech and audio processing, it has necessitated the building of specialist models for individual tasks and application scenarios. It is likewise difficult to apply this to dialects and languages for which only limited labeled data is available. Self-supervised representation learning methods promise a single universal model that would benefit a wide variety of tasks and domains. Such methods have shown success in natural language processing and computer vision domains, achieving new levels of performance while reducing the number of labels required for many downstream scenarios. Speech representation learning is experiencing similar progress in three main categories: generative, contrastive, and predictive methods. Other approaches rely on multi-modal data for pre-training, mixing text or visual data streams with speech. Although self-supervised speech representation is still a nascent research area, it is closely related to acoustic word embedding and learning with zero lexical resources, both of which have seen active research for many years. This review presents approaches for self-supervised speech representation learning and their connection to other research areas. Since many current methods focus solely on automatic speech recognition as a downstream task, we review recent efforts on benchmarking learned representations to extend the application beyond speech recognition.

I. INTRODUCTION

Self-supervised speech representation learning addresses the limitations of label-intensive specialist models by learning reusable representations from unlabeled audio. The review situates these methods within historical representation-learning research and a two-stage pre-training/downstream framework.

  • Supervised deep learning has driven major speech-processing gains but restricted progress in languages and domains with limited labeling investment.
  • Self-supervised methods learn representations from input data itself as labels, supporting downstream applications with minimal labeled data.The review describes this as a growing subcategory of unsupervised learning.
  • The framework separates SSL pre-training of a representation model from downstream use through frozen representations or supervised fine-tuning.ASR and speaker identification are examples of downstream applications.
  • Speech representations should disentangle factors such as speaker identity, style, emotion, and noise while remaining invariant to nuisance variation.
  • This review focuses exclusively on SSL for speech processing because speech differs substantially from image and text inputs and requires specialized theories and technologies.
  • The paper connects current neural approaches to earlier clustering, mixture-model, generative-feature, and deep-representation research.

C. Learning through pretext task optimization

Pretext-task optimization trains representations by making models solve challenging objectives on unlabeled speech. The review formalizes the representation pipeline and explains why speech-specific sequence, continuity, and task-diversity constraints complicate transfer from NLP and computer vision.

  • A well-designed pretext task must exploit unlabeled data while avoiding shortcuts that prevent learning high-level abstract representations.
  • SSL aims to learn task-agnostic representations, unlike pseudo-labeling, which uses task-specific supervised data during pre-training.
  • Speech pretext tasks cannot generally transfer directly from computer vision because speech is a variable-length sequence.
  • Long speech sequences lack obvious unlabeled segment boundaries, creating computational challenges for architectures such as Transformers.
  • Because speech is continuous and lacks a finite vocabulary, categorical masking objectives from NLP do not apply directly.
  • Different speech tasks require different information, such as content for recognition and speaker information for speaker recognition.
  • The representation function f(·) produces H from acoustic input X, while auxiliary functions support pretext training and can be discarded afterward.Downstream systems can fine-tune f(·) or use extracted features, sometimes from hidden layers.
  • Contextualized representations depend on long input subsequences spanning several phonemes, unlike localized representations based on short temporal neighborhoods.

B. Generative approaches

Generative approaches train speech representations by reconstructing original or future input from limited information. The category includes autoencoding, autoregressive prediction, and masked reconstruction, with quantization providing discrete representations for some models.

  • Generative pretext tasks reconstruct the original input from a limited view, including future prediction, masking, or corrupted-input reconstruction.This usage differs from generative models that learn distributions for sampling new data.
  • Autoencoding: Autoencoders use encoder–decoder architectures to reconstruct inputs, often imposing information bottlenecks or latent-space regularization.Denoising, variational, and vector-quantized variants extend the basic autoencoder framework.
  • Autoencoding: VQ-VAE quantizes encoder outputs through a codebook before an autoregressive decoder reconstructs the acoustic input.Auxiliary codebook and commitment losses keep encoder outputs and codebook vectors aligned.
  • Autoencoding: Discrete representations learned through vector quantization capture high-level speech information related to phonemes and support applications such as speaker conversion.
  • Autoregressive prediction: Autoregressive predictive coding predicts a future acoustic feature from past input representations using a learned encoder and projection module.The prediction horizon c is at least one timestep, and training minimizes an L1 loss.
  • Autoregressive prediction: APC uses only previous timesteps, whereas DeCoAR combines forward and backward encoders to represent information from the entire input.
  • Masked Reconstruction: Masked reconstruction learns contextualized speech representations by reconstructing masked regions from unmasked context.Speech masking policies vary, and Transformers are commonly used as encoders.
  • Masked Reconstruction: Phonetic-segmentation masking aligns masked regions with linguistic units but requires labeled data to obtain the segmentation.

3) Challenges:

Generative pretext tasks face unresolved design and representation challenges in speech. Reconstruction can encode redundant information, while masking and target choices influence learned representations but remain insufficiently studied.

  • Perfectly reconstructing speech may preserve redundant information instead of isolating features useful for a downstream task.Speech contains speaker identity and prosodic information beyond text, making generation more difficult than in text domains.
  • Masking strategy and the choice between waveform or spectral targets influence what a generative pretext task teaches the model.These design choices affect the information encoded in learned representations.
  • The relationship between generative task design and the information encoded in learned speech representations remains little studied.

C. Contrastive approaches

Contrastive approaches learn speech representations by distinguishing positive samples from negatives, often using contextual prediction, masking, or quantized targets. The reviewed models differ in architecture, sampling, and loss design, with wav2vec 2.0 combining contrastive learning and masking.

  • Contrastive models maximize similarity between an anchor and positive sample while minimizing similarity to distractor negatives.
  • CPC: CPC uses convolutional and recurrent modules to predict future localized representations from contextualized representations.Its loss compares one future positive with sampled negatives using a categorical classification objective.
  • wav2vec: wav2vec uses fully convolutional modules and treats each positive-negative comparison as an independent binary classification.Unlike CPC, its negative set excludes the target index because including it would cancel the positive term.
  • wav2vec 2.0: wav2vec 2.0 combines InfoNCE contrastive learning with masking, using contextualized masked representations and quantized localized representations.Its pipeline applies convolutional encoding, Transformer contextualization, masking, and quantization with Gumbel softmax and product quantization.
  • Reported impact: Wav2vec 2.0 first achieved single-digit WER on LibriSpeech using low-resource Libri-light subsets for fine-tuning.The approach subsequently inspired follow-up studies, including wav2vec-C, which adds a consistency term to reconstruct input features from quantized representations.

3) Challenges:

Contrastive speech representations are effective across downstream applications but face challenges caused by ambiguous sample definitions and the absence of explicit acoustic-unit segmentation.

  • Positive and negative sampling can impose unintended invariances, such as speaker invariance when negatives come exclusively from the same utterance.Whether that invariance is desirable depends on the downstream application.
  • Because speech lacks explicit acoustic-unit segmentation, sampled examples may contain partial or multiple language units.
  • Smooth speech signals make it difficult to define sampling strategies that reliably produce true positives and negatives.

D. Predictive approaches

Predictive approaches learn targets for pretext tasks without contrastive loss, typically using separate teacher models or externally computed targets. The reviewed methods include Discrete BERT, HuBERT, WavLM, and data2vec, with reported gains across speech and other modalities.

  • D. Predictive approaches: Predictive methods use learned targets with losses such as squared error or cross-entropy instead of contrastive loss.Targets are typically computed outside the model’s computational graph, often through a separate teacher model.
  • D. Predictive approaches: Discrete BERT converts continuous speech into discrete units with vq-wav2vec, then trains a BERT model to predict masked units.Its vq-wav2vec parameters remain frozen during BERT training.
  • D. Predictive approaches: 25% WER on test-other with 10 minutes of fine-tuning made Discrete BERT an early demonstration of self-supervised speech representation learning.The result helped establish the direction for subsequent approaches.
  • D. Predictive approaches: HuBERT predicts k-means-derived targets from raw waveforms and uses iterative clustering to improve target consistency and model representations.The procedure encourages acoustic modeling for unmasked frames and long-range temporal modeling for masked frames.
  • D. Predictive approaches: Two HuBERT pre-training iterations match or outperform previous state-of-the-art low-resource speech-recognition results, while w2v-BERT improves further by combining HuBERT and wav2vec 2.0.The reported result concerns low-resource speech recognition.
  • D. Predictive approaches: WavLM adds gated relative position bias and utterance mixing, while data2vec uses an EMA teacher and regression targets.WavLM improves phoneme and speech recognition performance and learns to filter overlapping speech; data2vec performs competitively across speech, images, and text.

3) Challenges:

Predictive speech SSL methods face practical and data-quality challenges. HuBERT and WavLM require iterative pre-training, while data2vec’s teacher-student design increases computational demands.

  • 3) Challenges: HuBERT and WavLM require iterative pre-training, which can be inconvenient when processing large volumes of data.The initial MFCC-derived vocabulary is another challenge for these models.
  • 3) Challenges: Data2vec continuously improves targets through an EMA teacher but increases the computational challenges of very large models.Mixed precision, model parallelism, or model sharding may be needed to reduce instantaneous memory use.

E. Learning from multi-modal data

Multimodal speech representation learning combines speech with complementary signals such as visual, articulatory, textual, or contextual data. Approaches use multi-view, contrastive, masked-prediction, or multimodal encoder designs to learn joint representations.

  • E. Learning from multi-modal data: Multimodal learning can reduce noise effects because noise in different modalities is often independent or uncorrelated.Accompanying images or video can also provide grounding signals that help encode more semantic information.
  • E. Learning from multi-modal data: Intrinsic modalities originate from the speech source and include faces, lip movements, articulatory measurements, and simultaneous MRI scans.These modalities are typically used to improve robustness to acoustic noise.
  • E. Learning from multi-modal data: Multimodal methods include multi-view autoencoders, deep Boltzmann machines, CCA, multi-view contrastive losses, and audio-visual masked prediction.Audio-Visual HuBERT is an example of the latter category.
  • E. Learning from multi-modal data: Extrinsic modalities come from different sources but provide context, such as an image paired with its spoken caption.The image can indicate what the speech is likely describing, supporting richer joint representations.
  • E. Learning from multi-modal data: Typical extrinsic approaches learn modality-specific encoders with multimodal contrastive, masked-margin-softmax, or masked-prediction losses.They are commonly evaluated on cross-modal retrieval and sometimes on ZeroSpeech and SUPERB tasks.
  • E. Learning from multi-modal data: SLAM combines separate pre-trained speech and text encoders with a multimodal encoder trained using two supervised and two self-supervised tasks.The approach builds joint speech-text representations from paired and unpaired data.

3) Challenges:

The review connects speech representation learning to segment-level embeddings and surveys datasets used for pre-training and evaluation. Key open issues include whether simple pooling yields useful word embeddings and how broadly multimodal representations transfer.

  • F. Acoustic Word Embeddings: Acoustic word embeddings represent variable-length speech spans such as phones, words, or phrases for similarity-based search.They have been built using template distances and neural autoencoders and are often evaluated through word discrimination.
  • 3) Challenges: Simple downsampling, mean pooling, and max pooling of self-supervised frame representations have not produced successful segment embeddings in initial results.More work is needed before drawing conclusive conclusions.
  • C. Datasets for evaluation: SSL pre-training datasets are usually large and have limited or no labels, including Libri-light with 60k hours of English audio.The review also describes multilingual, Chinese, labeled, and domain-specific corpora used for training or evaluation.
  • C. Datasets for evaluation: Multilingual corpora such as Common Voice, Multilingual LibriSpeech, VoxPopuli, and BABEL support large-scale pre-training and partial ASR evaluation.Common Voice is described as an open-source multilingual dataset spanning 76 languages and 11k hours at the review’s stated date.
  • C. Datasets for evaluation: Evaluation datasets cover ASR, phoneme recognition and classification, speaker tasks, sentiment, translation, query-by-example, intent classification, audio events, and language identification.The review lists Hub5, DIRHA, CHiME-5, INTERFACE, MOSEI, VCTK, VoxCeleb1, FSC, QUESST, LS En-Fr, CoVoST-2, ALFFA, and OpenSLR-multi.

D. Experiment settings for evaluating SSL techniques

The review compares SSL techniques across datasets, downstream tasks, transfer settings, and labeled-data regimes, focusing first on ASR using LibriSpeech test-clean. Results generally favor larger SSL models, especially when labels are scarce, but comparisons are confounded by differing training recipes.

  • Experiment settings: SSL evaluations vary pre-training corpora, downstream datasets and tasks, transfer-learning settings, and the amount of labeled fine-tuning data.Tables III and IV organize these experiment settings and indicate whether pre-training and fine-tuning use different datasets.
  • ASR benchmark: The survey focuses first on LibriSpeech ASR test-clean because the literature uses many diverse datasets and downstream tasks.It summarizes published WER after SSL pre-training and supervised fine-tuning with 960, 100, 10, 1, or 0.17 hours of labeled data.
  • Benchmark results: Most SSL techniques outperform fBANK features, and larger models generally achieve better performance as labeled resources decrease.The advantage of larger wav2vec 2.0 and HuBERT models becomes more evident in low-label settings.
  • Benchmark results: wav2vec 2.0 and HuBERT achieve lower or competitive WER than several semi-supervised methods with only 1 hour or 10 minutes of labels.The comparison includes iterative pseudo labeling, slimIPL, and noisy student using 100 hours of labels.
  • Beyond ASR: SSL models are competitive beyond ASR in IC, SID, ASV, and QbE, suggesting benefits across content, speaker, and semantic encoding tasks.Table V summarizes tasks where models with SSL pre-training achieve state-of-the-art results.
  • Caveats: Cross-paper results are less comparable because studies use different language models, prediction heads, and choices about freezing or fine-tuning encoders.Therefore, lower WER alone does not establish method superiority.
  • Benchmarking frameworks: SUPERB and LeBenchmark provide reproducible frameworks for evaluating SSL representations across diverse tasks and, in LeBenchmark, French speech applications.These benchmarks aim to reduce variation in downstream recipes and support broader, more reproducible comparisons.

V. ANALYSIS OF SELF-SUPERVISED REPRESENTATIONS

Analyses of self-supervised speech representations examine where acoustic, phonetic, speaker, language, and semantic information is encoded. They indicate that intermediate layers and the training criterion strongly shape downstream-relevant information, with trade-offs across task types.

  • Information content: Layer-wise analyses use CCA, probing tests, and related mechanisms to compare acoustic-linguistic information across representations and model layers.These analyses include pre-trained models of varying sizes and ASR-fine-tuned models.
  • Information content: Pre-trained models show an autoencoder-style pattern: early layers resemble input features, intermediate layers diverge, and final layers become more similar to inputs and early layers.Class-relevant information tends to peak in intermediate layers.
  • Information content: Lower WavLM layers encode speaker-related information, whereas higher layers encode phonetic and semantic information.This pattern was confirmed in a 95M-parameter model for speaker diarization and verification versus phonetic and semantic tasks.
  • Information content: Adding explicit self-supervised loss at HuBERT intermediate layers improves phonetic-content tasks but worsens speaker-related tasks.The reported improvements include phone recognition, ASR, and keyword spotting, while speaker diarization and verification decline.
  • Architecture analysis: Transformer attention heads specialize in neighboring frames, specific phonemes, or global context, with diagonal heads correlating strongly with phoneme boundaries.The analysis concerns generatively trained Transformer representation models.
  • Training criterion: Autoregressive predictive coding loss correlates most strongly with downstream performance on phone discrimination and speaker classification among the compared criteria.Comparisons included masked predictive coding, contrastive predictive coding, and autoregressive predictive coding.
  • Training criterion: The training criterion influences encoded information more than learner architecture or input directionality in the reported representation comparisons.This conclusion was drawn across models trained with masked, contrastive, and autoregressive predictive criteria.
  • Training criterion: Compared with vq-vae, vq-wav2vec uses contextual prediction of future latent discrete representations rather than reconstructing audio from discrete latents.The passage frames the comparison around discovering phonetic units.

C. Effects of data and model size

Larger datasets and models generally improve downstream performance, while model accuracy eventually saturates. Self-supervised representations also transfer across languages and benefit from unlabeled data, including in low-resource settings.

  • Data size: Larger training datasets improve downstream performance, as shown by comparisons involving 960 and 8,000 hours of speech.The cited comparison uses representations learned by a bidirectional CPC model.
  • Model size: Model size and downstream accuracy are positively correlated for phone classification and speaker recognition, but accuracy does not exactly follow a power law and eventually saturates.Self-supervised L1 loss approximately follows a power law, unlike the reported downstream accuracies.
  • Robustness and transferability: 66–73% of the performance gap is closed by out-of-domain unlabeled data relative to in-domain labeled data and a supervised out-of-domain model.The result concerns robustness under distributional shifts and compares pre-trained representations with MFCCs.
  • Cross-lingual transfer: English-only CPC pre-training enabled phone discrimination in 10 other languages, rivaling supervised methods with 1h of labeled data per language.Multilingual wav2vec 2.0 pre-training also improved character-based ASR, especially for low-resource languages.
  • Unsupervised ASR: Unsupervised ASR methods can learn from unpaired speech and text, but speech-to-text mapping must handle variable-length acoustic segments and unavailable token boundaries.Common systems use phoneme segmentation before the generator; wav2vec-U 2.0 removes that segmentation stage.
  • Unsupervised ASR: 11.3% phoneme error rate was achieved on TIMIT, while wav2vec-U achieved 5.9% WER on LibriSpeech test-other using unpaired data.The reported wav2vec-U results were state of the art at the time; domain mismatch still reduces performance.

2) ASR-TTS:

ASR-TTS methods use cascaded speech recognition and synthesis systems to learn from unpaired audio and text, while zero-resource technologies extend learning to audio without linguistic resources. These approaches support broader evaluation across languages, speakers, synthesis, and language modeling.

  • ASR-TTS: ASR-TTS frameworks combine ASR and TTS in a cascaded autoencoder-like system, treating the intermediate ASR output as a latent representation.The ASR encoder maps speech to text-like representations, while the TTS decoder reconstructs acoustic features.
  • ASR-TTS: ASR-TTS systems can jointly optimize ASR and TTS using unpaired audio and text, but practical systems often require paired data and are categorized as semi-supervised.The paired-data requirement is identified as a condition for making the complicated system work.
  • ASR-TTS: Text-only resources can be paired with synthetic speech generated by multi-speaker TTS or by generating higher-level linguistic features.These synthetic pairs support ASR training without corresponding real audio data.
  • Zero-resource speech: Zero-resource speech technologies discover acoustic and linguistic units from audio without text, lexicons, transcriptions, or other linguistic annotations.The field includes acoustic and linguistic unit discovery and is evaluated through successive zero-resource challenges.
  • Zero-resource speech: Zero-resource challenges expanded from subword modeling and spoken-term discovery to unseen languages and speakers, speech synthesis, word-like unit discovery, and language modeling.The 2021 challenge added lexical, semantic, and syntactic metrics computed from pseudo-acoustic labels.

2) Textless NLP:

Textless NLP models language directly from audio by using discrete speech representations instead of text or labels. The review identifies this direction as promising while noting unresolved challenges in semantic information, efficiency, data requirements, robustness, and representation use.

  • Textless NLP: Textless NLP models language directly from audio, bypassing text and labels while retaining non-lexical information such as speaker identity, emotion, hesitation, and interruptions.The GSLM uses discrete representations from wav2vec 2.0, HuBERT, and CPC as inputs to an autoregressive language model.
  • Discussion and conclusion: The review identifies wav2vec 2.0 as a widely adopted standard model and HuBERT as a simple, stable basis for extensions in representation quality, speech translation, and textless NLP.Wav2vec 2.0's multilingual variants contributed to its adoption, while HuBERT resembles classic frame-level ASR training.
  • Future directions: Adapters and prompt or instruction learning offer parameter-efficient alternatives to fine-tuning, but speech research remains limited and prompts underperform on sequence generation tasks.The review identifies prompt use for phoneme recognition and slot filling as an open question.
  • Future directions: Larger representation models improve downstream performance but increase memory and time costs, limiting edge deployment and scaling to very large datasets.The review also calls for more data-efficient approaches because current methods require large volumes of unlabeled speech.
  • Future directions: Further work targets feature disentanglement and robustness, while failure modes and relative robustness to adversarial attacks remain poorly understood.One proposed decomposition separates content and speaker embeddings for ASR and speaker identification.
  • Future directions: Existing speech representations still struggle with higher-level semantic tasks that word-level models such as BERT capture more easily.Two-stage training is one workaround, but it prevents semantic knowledge from the second stage benefiting the phonetically focused first stage.
  • Future directions: Speech corpora contain far less content information than text corpora, motivating the use of text representation models to improve speech representations.Matching BERT's corpus content would require an estimated 400,000 hours of audio at 120 words per minute.
Loading 2205.10643v3…