PaperlayerView exact revision on arXiv

Source-linked AI summary

Audio Self-supervised Learning: A Survey

Shuo Liu, Adria Mallol-Ragolta, Emilia Parada-Cabeleiro, Kun Qian, Xin Jing, Alexander Kathan, Bin Hu, Bjoern W. Schuller

arXiv:2203.01205v1cs.SDcs.AIeess.AS

TL;DR

Audio SSL research lacked systematic investigation and a unified overview. This survey reviews audio SSL approaches and their performance across classic audio tasks, reporting that methods such as Wav2Vec 2.0 have surpassed supervised methods on the same task.

  • Problem

    As audio SSL research grew, the area lacked systematic investigation and a unified overview of its approaches.

    Source: §I · p. 2

  • Method

    The survey overviews uni-modal and multi-modal audio SSL approaches and analyses their performance across classic audio tasks.

    Source: §I · p. 2, §VII · p. 21

  • Results

    Across several classic audio tasks, the survey reports successful SSL performance and notes that Wav2Vec 2.0 surpassed supervised methods on the same task.

    Source: §III · p. 12, §VII · p. 21

  • Takeaways & Limitations

    The survey concludes that SSL performance supports representation generalisation and can reduce the urgency of searching for hand-crafted engineered features.

    Source: §VII · p. 21

Abstract

from arXiv · show

Inspired by the humans' cognitive ability to generalise knowledge and skills, Self-Supervised Learning (SSL) targets at discovering general representations from large-scale data without requiring human annotations, which is an expensive and time consuming task. Its success in the fields of computer vision and natural language processing have prompted its recent adoption into the field of audio and speech processing. Comprehensive reviews summarising the knowledge in audio SSL are currently missing. To fill this gap, in the present work, we provide an overview of the SSL methods used for audio and speech processing applications. Herein, we also summarise the empirical works that exploit the audio modality in multi-modal SSL frameworks, and the existing suitable benchmarks to evaluate the power of SSL in the computer audition domain. Finally, we discuss some open problems and point out the future directions on the development of audio SSL.

I. INTRODUCTION · II. SELF-SUPERVISED LEARNING: A GENERAL OVERVIEW · A. Views Generation

The paper frames SSL as learning generalisable representations from data through pretext tasks, then applying them to downstream tasks. It surveys audio SSL and explains how views are generated to support self-supervised learning.

I. INTRODUCTION

  • Inspired by cognitive development and transfer of learning, SSL learns representations from data attributes through pretext tasks, then applies them to new contexts in downstream tasks.The analogy is that new knowledge is integrated with prior knowledge to support generalisation.Source: §I · p. 1
  • By learning without human labels, SSL can reduce annotation burdens and task-specific architecture design, while negative examples can make representations more distinctive, –.Upstream representations can serve multiple downstream tasks, and incorporating upstream models can reduce the labelled data needed for fine-tuning.Source: §I · pp. 1–2
  • Audio SSL faces substantial variability, overlapping noise, and a need for representations that are distributed, abstract, and disentangled, motivating a survey of this under-reviewed area.Earlier SSL surveys were biased toward computer vision and NLP; this work presents an overview of audio SSL within unified frameworks.Source: §I · p. 2

II. SELF-SUPERVISED LEARNING: A GENERAL OVERVIEW

  • SSL creates correlated views and learns their representations; frameworks are predictive when they omit negative samples and contrastive when they use them.The overview also identifies contrastive predictive coding, based on autoregressive and masked predictive coding, as a predictive-model variant that can use contrastive loss.Source: §II · p. 2

A. Views Generation

  • Views can come from multiple sensors or paired modalities, such as speech with transcripts or audio with video frames –.The paper notes that multimodal and cross-modal SSL are common and treats audio-specific examples in its later discussion.Source: A. Views Generation · pp. 2–3
  • Audio views can be generated by transforming spectrograms or Mel features with warping and SpecAugment –, or by pairing segments from the same recording –.For sequential signals, nearby segments can form positive pairs, while distant segments or segments from other recordings can serve as negatives.Source: A. Views Generation · p. 3
  • Global-local view methods maximise mutual information between a context vector and local features; autoregressive temporal context with contrastive loss underlies CPC, [40].CPC can learn fixed-length audio representations from speech input.Source: A. Views Generation · p. 3
  • View selection also considers redundancy: suggests using positive pairs with less mutual information, while reports contrastive loss is not solely attributable to mutual information.The proposed aim is to compress redundancy in view embeddings that is irrelevant to labels.Source: A. Views Generation · p. 3

B. SSL Frameworks · 1) Predictive Models:

Predictive SSL learns representations by predicting or reconstructing input information, aligning views of the same sample, or assigning pseudo-labels through clustering. This section describes auto-encoding, Siamese, and clustering frameworks and their audio and speech applications.

1) Predictive Models:

  • Auto-encoders reconstruct inputs from compressed latent embeddings, but oversized latent spaces can permit identity mapping; denoising and other augmentations help prevent it.Denoising auto-encoders corrupt input values and learn to recover the undistorted input; prediction may instead target only the distorted part.Source: 1) Predictive Models · pp. 3–4
  • CBoW predicts a word from context and is stronger on syntax but prone to frequent-word overfitting, while Skip-gram predicts context and better captures semantics; both inform audio-segment embeddings.Audio2Vec and Speech2Vec apply these formulations to audio or speech transformed into time and Mel-frequency representations.Source: 1) Predictive Models · pp. 4–5
  • APC predicts future audio representations from past context, whereas MPC -inspired methods mask inputs for bidirectional prediction and train faster; NPC uses local dependencies.APC uses an RNN context network and L1 loss, while MPC can use Transformer encoders or bidirectional RNNs; other predictive tasks predict relative position or reorder sequence parts,,,,,,.Source: 1) Predictive Models · p. 5
  • Without negative samples, Siamese models risk mode collapse, yet BYOL, SimSiam, and Barlow Twins use positive pairs alone and retain competitive or superior SSL performance.Siamese architectures process two views in separate towers and use one tower’s representation as a pseudo-label for the other.Source: 1) Predictive Models · p. 5
  • BYOL uses an online predictor and a slowly updated target network, while SimSiam removes projection layers and copies the target from the online network.BYOL updates the target by exponential moving average, uses symmetric losses across swapped views, and applies LARS to help avoid collapse.Source: 1) Predictive Models · pp. 5–6
  • Barlow Twins trains two identical networks to make their output cross-correlation matrix approach identity, combining view invariance with reduced redundancy.Its design avoids BYOL and SimSiam’s asymmetric components, gradient stopping, and EMA, and benefits from high-dimensional output vectors.Source: 1) Predictive Models · p. 6
  • Deep Cluster alternates K-means pseudo-labeling with encoder classification, Local Aggregation optimizes local soft clustering, and SwAV performs online clustering with multi-crop swapped prediction.SwAV assigns pseudo-labels within mini-batches, avoiding the two-step training paradigm.Source: 1) Predictive Models · pp. 6–7

2) Contrastive Models:

Contrastive SSL learns representations by bringing positive pairs together and separating negative pairs, using losses and Siamese architectures that manage negative samples. Its effectiveness depends on balancing embedding alignment, uniformity, and tolerance for semantically similar inputs.

Contrastive loss

  • Max-margin contrastive loss pulls same-class embeddings together and separates different-class pairs, while triplet loss sets positive-negative offsets and N-pair loss jointly compares multiple negatives.The account describes these objectives as progressing from pairwise comparisons to joint comparisons among multiple negatives.Source: 2) Contrastive Models · p. 7
  • NT-Xent adds a temperature parameter to control negative-sample penalties, while InfoNCE [40], inspired by Noise Contrastive Estimation, contrasts one positive against N −1 negatives as an N-class cross-entropy task.The classifier assigns larger values to the positive example than to negative noise samples.Source: 2) Contrastive Models · p. 7

Contrastive SSL for Siamese Models

  • SimCLR uses augmented views, a ResNet encoder, projection heads, and NT-Xent, with success attributed to scaling batch size, network depth and width, and training duration.Its described augmentations include cropping, resizing, colour distortion, and Gaussian blur.Source: 2) Contrastive Models · p. 7
  • MoCo uses a momentum encoder and queued prior negatives; MoCo v2 [44] strengthens augmentation, SimCLR v2 scales up with memory, and MoCo v3 drops the queue for larger batches.MoCo’s momentum encoder updates without back-propagation; v2 expands projection heads, SimCLR v2 retains a layer for few-label fine-tuning, and v3 adds a prediction layer.Source: 2) Contrastive Models · pp. 7–8
  • When two views contain redundant label information, linear projections of learned representations guarantee downstream prediction performance; contrastive negatives also prevent collapse by balancing positive-pair alignment and embedding uniformity.Uniformity distributes normalized embeddings across a unit sphere, and both alignment and uniformity are reported as important for downstream tasks.Source: 2) Contrastive Models · p. 8
  • A temperature that is too small removes tolerance for semantically similar samples, can damage their underlying semantic structure, and harms downstream features; temperature therefore balances separability and tolerance.The passage relates temperature’s effect to the margin value in the earlier contrastive-loss formulation.Source: 2) Contrastive Models · p. 8
  • Wang proposes adjusting alignment and uniformity losses so both are minimized simultaneously, reporting that their balance is more effective than standard contrastive loss.The adjustment targets maintaining a good balance between the two losses.Source: 2) Contrastive Models · p. 8

3) Contrastive Predictive Coding:

CPC uses an autoregressive model to predict future information from past-frame context, with negative samples supporting contrastive representation learning. Related approaches extend contrastive loss to masked predictive models and wav2vec 2.0.

3) Contrastive Predictive Coding:

  • CPC [40] predicts future information from aggregated past-frame context and uses negative samples in an InfoNCE objective to improve representation discrimination.The objective replaces the query data point x with a context vector c_t.Source: 3) Contrastive Predictive Coding · p. 8
  • CPC v2 increases model capacity, replaces batch normalization with layer normalization, and adds patch-based augmentation.Batch normalization was found to harm downstream tasks in CPC frameworks.Source: 3) Contrastive Predictive Coding · p. 8
  • Contrastive loss is also used in masked predictive models, producing contrastive MPC.The passage states that its training objective can be formalized but supplies no further details here.Source: 3) Contrastive Predictive Coding · p. 8
  • wav2vec 2.0 modifies this loss and generates negative samples in the same way as CPC.Source: 3) Contrastive Predictive Coding · p. 8

C. Training with or without negative samples

The section contrasts non-contrastive SSL, which may preserve more complete information, with contrastive SSL, which may learn more discriminative features. It also notes that training objectives and negative-sample quantity and quality affect SSL performance.

C. Training with or without negative samples

  • Training objectives matter more than network architecture, and increasing model and representation size can improve learned representations.For contrastive learning, the quantity and quality of negative samples also affect performance.Source: C. Training with or without negative… · pp. 8–9
  • Non-contrastive SSL aims to bring representations of views from the same data point together, while contrastive SSL separates positive from negative samples.The section argues that non-contrastive methods may preserve salient information, whereas contrastive methods may discard shared attributes that are less useful for distinguishing samples; it infers that non-contrastive SSL may therefore be more useful,.Source: C. Training with or without negative… · p. 9
  • Contrastive SSL may perform better when its pretext and downstream tasks are closely matched, such as reading-speech pretraining for speech recognition.Fine-tuning can help contrastive models capture more complete representations, but the section says reducing redundancy appears harder than restoring lost completeness,.Source: C. Training with or without negative… · p. 9

III. AUDIO SSL

Audio SSL encompasses contrastive, predictive, masked-reconstruction, and task-specific approaches that learn representations from speech or general audio. The section describes methods for ASR and other downstream tasks, as well as speech enhancement and source separation.

Contrastive methods

  • Audio contrastive methods extend SimCLR across raw audio and time-frequency inputs, while BYOL-A learns from a single audio example without negative samples, –,,.CLAR reports that combining a contrastive loss with supervised training can improve convergence speed and representation effectiveness using substantially less labelled data; Wang maximises agreement between waveform and spectral representations.Source: §III · p. 9

Other pretext tasks

  • Audio2Vec and Speech2Vec adapt CBoW and skip-gram objectives to audio, while other pretext tasks predict temporal gaps or reorder shuffled spectrogram patches,,.Audio2Vec requires no explicit assistance, whereas Speech2Vec uses forced alignment; permutation inversion can be trained end-to-end as a classification task.Source: §III · pp. 9–10

Masked acoustic models

  • Masked acoustic models learn representations by reconstructing masked audio, predicting only missing spectrogram components, or jointly solving multiple prediction tasks –[97].Mockingjay reconstructs complete Mel-spectrogram frames; DAPC predicts missing time and frequency components, while PASE combines a CNN encoder with multiple task-specific workers.Source: §III · pp. 10–11
  • TERA outperforms Mockingjay and Audio ALBERT on phoneme classification, keyword spotting, and speaker classification, with promising ASR results on Librispeech and TIMIT.Its transformer pretraining extends masking to contiguous segments, channel-axis masking, and Gaussian noise.Source: §III · p. 10

Wav2vec models

  • CPC predicts future latent representations contrastively, and later wav2vec methods adapt this framework with convolutional or quantised representations [40],,,.CPC contrasts true future samples with noise; wav2vec 2.0 instead masks encoded audio and uses contextualised representations in its contrastive objective.Source: §III · pp. 11–12
  • Wav2vec 2.0 achieves state-of-the-art results on ASR evaluations using Librispeech and TIMIT,.It uses bidirectional masked prediction, contrastive learning, and product quantisation.Source: §III · p. 12
  • Wav2vec variants extend to domain adaptation, noisy far-field ASR, and unsupervised phoneme learning, while HuBERT uses K-means-derived pseudo-labels rather than contrastive learning,, [99],, [115],.Pretraining across multiple domains can improve generalisation; Wav2vec-C adds a VQ-VAE reconstruction objective and improves ASR on real-world far-field noisy data over Wav2vec 2.0.Source: §III · pp. 12–13

Enhancement and separation

  • Audio SSL also targets speech enhancement and source separation through mixture-to-clean mappings, MixIT, denoising pretraining, and speaker-specific feature learning [120]–,.Denoising pretraining captures speech structure and relaxes the permutation problem in source separation; PseudoSE and contrastive mixtures instead learn speaker-specific features from premixture signals and require downstream fine-tuning.Source: §III · pp. 13–14

IV. MULTI-MODAL AUDIO REPRESENTATION

Multi-modal SSL uses complementary modalities as different views of one object to improve their representations, and this section surveys approaches that include audio, especially audio-visual methods.

IV. MULTI-MODAL AUDIO REPRESENTATION

  • Audio-visual SSL commonly learns correspondence between video frames and audio, using synchronized streams as inputs to a Siamese network where each modality supervises the other.The approaches treat the visual and audio streams as two views.Source: §IV · p. 14
  • Other approaches learn audio representations through video generation, object segmentation or localisation, source separation, and speech-text similarity based on linguistic structure.For video generation, each utterance segment's representation is expected to carry adequate speech information for transferring knowledge into video frames.Source: §IV · p. 14

A. Audio & Visual · 1) Visual-Audio Correspondence Decision:

The subsection surveys audio-visual self-supervision through correspondence, contrastive pairing, and cross-modal supervision, including methods for localizing sound sources and learning representations for downstream tasks.

1) Visual-Audio Correspondence Decision:

  • L3-Net and AVE-Net classify whether a one-second audio segment and its corresponding video frame match, using concatenation or Euclidean distance, respectively.Both are designed to localize sound sources: L3-Net’s vision network recognizes sound-producing entities, while AVE-Net compares audio with spatial visual grids; both use logistic loss.Source: 1) Visual-Audio Correspondence Decision · p. 14, Figure 8 · p. 15
  • Synchronization learning extends correspondence to video-frame sequences with contrastive objectives, while curriculum learning gradually increases pair difficulty for initialization and convergence.An earlier study also reported success using synchronously recorded ambient sounds to supervise visual learning.Source: 1) Visual-Audio Correspondence Decision · pp. 14, 16
  • Because easy negatives can dominate learning, ultrasound work starts with hard pairs, while a teacher-student curriculum alternates frozen audio and vision teachers and uses a memory bank.The memory bank enlarges the negative set and improves action recognition and audio sound recognition.Source: 1) Visual-Audio Correspondence Decision · p. 15
  • Margin-based pairing contrasts equal numbers of positive and negative examples, including easy, hard, and super-hard negatives; staged difficulty also supports diarization and disentangled representations, –.For, negatives range from different videos to temporally distant or overlapping pairs; diarization adjusts triplet-loss margins by stream offset, while, separate content and identity representations.Source: 1) Visual-Audio Correspondence Decision · pp. 15–16, Figure 9 · p. 15
  • Spoken-caption–image methods learn shared representations through similarity or triplet objectives, enable pixel-level localization, and can align discrete linguistic units with visual representations –.[139] contrasts matched pairs with unmatched captions or images; uses ResDAVEnet-VQ units and Tacotron2 speech synthesis rather than contrastively reproducing audio.Source: Figure 8 · p. 15, 1) Visual-Audio Correspondence Decision · p. 16
  • Cross-modal pseudo-labeling outperformed single-modality SSL and pseudo-labeling from both modalities on action recognition, using clustering or memory-feature contrastive learning,.[144] additionally uses Cross-Modal Agreement to calibrate within-modal similarities between positive pairs.Source: 1) Visual-Audio Correspondence Decision · p. 16, Figure 10 · p. 16

2) Audio-Visual Source Separation:

Audio-visual source separation methods use visual context and audio-video correspondence to separate or represent sound sources. The approaches progress from jointly learning visual segmentation and separation to modeling source-specific audio-visual objects.

2) Audio-Visual Source Separation:

  • PixelPlayer jointly learns visual object segmentation and audio source separation, conditioning separation on video frames; synchronized audio-visual data is later required to disentangle representations for independent use.Its three networks analyze video, mixture audio, and synthesize separated sources; the learned audio representations alone cannot separate audio from each video.Source: 2) Audio-Visual Source Separation · pp. 16–17
  • AudioScope pools embeddings of separated audio, attends between global audio and local spatial-temporal video features, then concatenates global audio and video embeddings to form a MixIT assignment.Source: 2) Audio-Visual Source Separation · p. 17
  • LWTNet detects and tracks discrete audio-visual objects by learning fine-grained synchrony attention, contrasting each true audio track against shifted, misaligned audio segments.It extracts an embedding for each object and maximizes similarity between video frames and aligned audio while minimizing similarity to misaligned audio.Source: 2) Audio-Visual Source Separation · p. 17

3) Video Generation:

Video-generation methods use a starting speaker frame and its speech utterance to generate subsequent video sequences. The surveyed approaches also support spoken-word classification, lip reading, and emotion recognition.

Video Generation

  • The methods encode a starting frame with a U-Net and speech with an audio encoder, then combine their representations to generate subsequent video frames –.In, neural decoders also reconstruct audio features, and L1 errors for both audio and video outputs are minimized.Source: 3) Video Generation · p. 17
  • In, audio decoders are omitted, input audio is shortened to 0.2 s, and appended Gaussian noise injects randomness into face synthesis.The approaches in and demonstrate efficacy for spoken-word classification and lip reading.Source: 3) Video Generation · p. 17
  • A later approach by the same authors extends video generation to emotion recognition.Source: 3) Video Generation · p. 17

B. Audio & Text · C. Audio & Text & Video · V. DOWNSTREAM AUDIO TASKS & BENCHMARKS

The sections survey methods for aligning audio with text and video, then describe downstream tasks and benchmarks used to evaluate audio SSL representations.

B. Audio & Text

  • Wav2vec-U segments unlabelled speech and maps it to phonemes adversarially; Chung et al. align Speech2Vec and Word2Vec spaces for low- or zero-resource ASR and speech translation.Their method learns a linear mapping between the speech and text embedding spaces through adversarial training.Source: B. Audio & Text · p. 17
  • COALA contrastively aligns reconstructed audio and text semantic features; reconstructs spectrograms while predicting linguistic features, and CSTNet achieves comparable ABX and phone-recognition results.The auto-encoder reconstructions regularise representations to retain salient input information; COALA jointly optimises reconstruction errors and contrastive loss.Source: B. Audio & Text · pp. 17–18

C. Audio & Text & Video

  • In keyword localisation, the visually supervised model performs worse than text supervision based on BoW; it sometimes locates semantically related words, but not consistently [155].Source: C. Audio & Text & Video · p. 18
  • [151] FAC uses fine-grained audio-vision and coarser three-modality alignment; VATT aligns audio and video with NCE before adding text via MIL-NEC, using transformers for all modalities.FAC accounts for differences in granularity between visual/audio and language domains, with no direct audio-text link.Source: C. Audio & Text & Video · p. 18

V. DOWNSTREAM AUDIO TASKS & BENCHMARKS

  • Downstream tasks assess whether audio SSL representations generalise and discriminate; ASR evaluates Wav2vec methods,,, and public benchmarks support fair comparisons.Other evaluation tasks include speaker identification, emotion recognition, speech translation, pitch detection, and acoustic scene classification.Source: §V · p. 18
  • ZeroSpeech covers unsupervised subword modelling and spoken-term discovery, then expands to language and speaker variation, label-free speech synthesis, and speech-only or visually grounded language modelling.The 2019 challenge asked participants to discover subword units from raw audio and align them to recordings for synthesising target speakers.Source: §V · pp. 18–19
  • SuperB provides ten downstream speech tasks; LeBenchmark evaluates French SSL models on ASR, SLU, speech translation, and emotion recognition.LeBenchmark provides pretrained models learned from heterogeneous French read, prepared, and spontaneous speech.Source: §V · p. 19
  • Libri-Light benchmarks ASR with limited or no supervision using LibriVox English audiobooks; HEAR evaluates holistic speech and non-speech representations, including word classification, pitch detection, and sound-event detection.Source: §V · p. 19

VI. DISCUSSION · A. Difference from Other Confusing Learning Mechanisms

The discussion distinguishes audio SSL from related learning mechanisms and identifies common development challenges, including audio-specific differences in data processing, augmentation, negative-sample generation, and network construction. It also explains how representation and contrastive SSL relate to supervised, unsupervised, metric, and instance-discrimination learning, and notes GAN-based approaches as a future direction for audio SSL.

VI. DISCUSSION

  • Audio SSL development faces common difficulties and modality-specific concerns involving data processing and augmentation, negative-sample generation, and network construction.The discussion first frames these challenges alongside its comparison of SSL with other machine-learning mechanisms.Source: §VI · p. 19

A. Difference from Other Confusing Learning Mechanisms

  • Representation SSL learns representations by solving pretext tasks on large-scale unannotated data, unlike supervised transfer learning and domain adaptation or clustering-based unsupervised learning.Its automatically created targets provide supervision, but no labels from the target task are used; useful representations should retain salient information while generalising and distinguishing data.Source: A. Difference from Other Confusing… · p. 19
  • Contrastive SSL resembles metric learning because it brings positive pairs together and separates negatives, with positive pairs often created from the same data source through augmentation.The passage characterizes this as a relationship between contrastive SSL and weakly supervised metric learning.Source: A. Difference from Other Confusing… · p. 19
  • Instance discrimination assigns each sample its own class, and contrastive SSL can have a similar effect when InfoNCE uses a temperature that is too small.At that temperature, the two inputs in a positive pair tend to be treated as different instances.Source: A. Difference from Other Confusing… · p. 19
  • GANs have been treated as SSL frameworks,,: a generator creates data from a random vector using real data as targets, while a discriminator measures similarity to real data.That similarity measure changes as the discriminator is updated.Source: A. Difference from Other Confusing… · p. 19
  • GAN-based generative contrastive models have been explored for NLP, such as ELETRA, but rarely for audio SSL, making them a potential future direction.The survey did not include this form in its audio SSL literature review because it had rarely been explored in audio, while noting it should be considered in future work.Source: A. Difference from Other Confusing… · pp. 19–20

B. Difficulties and Problems for SSL Optimisation … VII. CONCLUSION

The survey concludes that audio SSL learns general representations across audio tasks and can outperform supervised methods, while its effectiveness depends on objective design, audio-specific modeling, and fit between pretext and downstream tasks.

B. Difficulties and Problems for SSL Optimisation

  • SSL representation quality depends on pretext-task objectives: predictive and contrastive methods align two views of one sample, while contrastive methods also separate other samples.These objectives focus on correlations between representations of observed data.Source: B. Difficulties and Problems for SSL… · p. 20
  • Predictive SSL can collapse, while contrastive learning mitigates collapse but faces unresolved trade-offs in negative sampling and early degeneration.Predictive Siamese networks commonly use asymmetric architectures, asynchronous updates, and augmented views; contrastive methods promote uniform representations on a unit sphere, and InfoNCE approximately maximises mutual information. More negatives raise the mutual-information lower bound, [40] and can improve representations,, as in audio SSL [40],,, but require more memory; too many may hurt training, no standard setting exists, and early overfitting to the pretext task can impair generalisation.Source: B. Difficulties and Problems for SSL… · p. 20

C. Additional adjustments on SSL for Audio

  • Audio SSL adapts CV methods by treating spectrograms, Mel-spectrograms, or MFCCs as images and applying CV augmentations, or by processing features as sequential frames for NLP methods,,,,.Source: C. Additional adjustments on SSL for… · p. 20
  • Direct waveform encoders, including 1D convolutions, provide another audio SSL route [40],,, but architecture effects remain unclear and warrant evaluation, including transformer attention.Audio SSL researchers often choose architectures designed for speech or audio structure, which may be more promising in this context; the survey focuses on SSL frameworks and formulations rather than architecture assessment.Source: C. Additional adjustments on SSL for… · p. 20

D. Fitness and mismatch between pretext and downstream tasks

  • Speech-pretrained SSL may transfer to acoustic scene classification, but scene-pretrained models may be weak on advanced speech tasks such as ASR.The expected source–target match can fail because speech varies more across time and frequency, whereas acoustic scene recordings are usually more temporally stationary.Source: D. Fitness and mismatch between… · p. 20
  • A small amount of downstream labels can guide SSL pretraining, and jointly optimising SSL and supervised objectives can improve speech representations; audio SSL has reached or surpassed supervised state-of-the-art results.This semi-supervised approach aims to close the gap between source and target domains; examples with available or partial labels include CLAR and UniSpeech.Source: D. Fitness and mismatch between… · pp. 20–21

VII. CONCLUSION

  • Audio SSL methods, including Wav2Vec 2.0, have surpassed supervised methods on some tasks, and their generalising representations can reduce reliance on hand-crafted features.The survey reviews uni-modal and multi-modal approaches across speech recognition, speaker identification, speech emotion recognition, and acoustic scene classification, and argues that SSL can advance audio processing.Source: §VII · p. 21
Loading 2203.01205v1…