PaperlayerView exact revision on arXiv

Source-linked AI summary

SSAST: Self-Supervised Audio Spectrogram Transformer

Yuan Gong, Cheng-I Jeff Lai, Yu-An Chung, James Glass

arXiv:2110.09784v2cs.SDcs.AIeess.AS

TL;DR

AST delivers strong audio classification performance but requires more data and depends on supervised pretraining with substantial labeled data. The paper introduces SSAST, which pretrains AST using joint discriminative and generative masked spectrogram patch modeling on unlabeled AudioSet and Librispeech audio. Across six downstream benchmarks, SSAST improves AST by an average of 60.9% and matches or outperforms supervised pretrained models.

  • Problem

    AST performs poorly from scratch and relies on supervised pretraining, while pure self-attention models generally require more training data than CNNs.

  • Method

    SSAST pretrains AST with joint discriminative and generative masked spectrogram patch modeling using unlabeled AudioSet and Librispeech data.

  • Results

    SSAST significantly improves AST on all six evaluated benchmarks, with an average improvement of 60.9%, and can match or outperform supervised pretrained models.

  • Takeaways & Limitations

    MSPM can replace supervised pretraining requiring large amounts of labeled data while supporting generalization across audio and speech classification tasks.

  • Takeaways & Limitations

    Frozen wav2vec and HuBERT can match or outperform fine-tuned SSAST for speech tasks, but they used substantially more GPUs and larger batch sizes.

Abstract

from arXiv · show

Recently, neural networks based purely on self-attention, such as the Vision Transformer (ViT), have been shown to outperform deep learning models constructed with convolutional neural networks (CNNs) on various vision tasks, thus extending the success of Transformers, which were originally developed for language processing, to the vision domain. A recent study showed that a similar methodology can also be applied to the audio domain. Specifically, the Audio Spectrogram Transformer (AST) achieves state-of-the-art results on various audio classification benchmarks. However, pure Transformer models tend to require more training data compared to CNNs, and the success of the AST relies on supervised pretraining that requires a large amount of labeled data and a complex training pipeline, thus limiting the practical usage of AST. This paper focuses on audio and speech classification, and aims to reduce the need for large amounts of labeled data for AST by leveraging self-supervised learning using unlabeled data. Specifically, we propose to pretrain the AST model with joint discriminative and generative masked spectrogram patch modeling (MSPM) using unlabeled audio from AudioSet and Librispeech. We evaluate our pretrained models on both audio and speech classification tasks including audio event classification, keyword spotting, emotion recognition, and speaker identification. The proposed self-supervised framework significantly boosts AST performance on all tasks, with an average improvement of 60.9%, leading to similar or even better results than a supervised pretrained AST. To the best of our knowledge, it is the first patch-based self-supervised learning framework in the audio and speech domain, and also the first self-supervised learning framework for AST.

1 Introduction

Pure self-attention models achieve strong audio results but require substantial data, while AST depends on supervised pretraining. SSAST addresses this limitation by learning from unlabeled audio with joint masked spectrogram patch objectives and evaluates across speech and audio tasks.

  • AST achieves state-of-the-art audio classification results but performs poorly when trained from scratch and relies strongly on supervised pretraining.
  • Unlabeled audio and speech data from sources such as radio or YouTube provide a basis for reducing AST’s labeled-data requirement.
  • MSPM jointly predicts masked spectrogram patches discriminatively and reconstructs them generatively, targeting both temporal and frequency structure.
  • The model is pretrained on Librispeech and AudioSet and evaluated on audio event classification, keyword spotting, speaker identification, and speech emotion recognition.
  • MSPM improves all six evaluated benchmarks by an average of 60.9% and can match or outperform supervised pretrained models.

2 Self-Supervised Audio Spectrogram Transformer

SSAST retains the AST Transformer architecture while replacing conventional pretraining with patch-level masking and joint discriminative-generative objectives. Randomly clustered masking exposes local and global spectrogram structure, and masked outputs are used for classification and reconstruction.

  • AST Model Architecture: The AST converts log Mel filterbank spectrograms into non-overlapping 16×16 patches, projects them into embeddings, adds positional embeddings, and processes them with a Transformer encoder.
  • AST Model Architecture: During fine-tuning and inference, mean pooling over Transformer patch representations produces an audio-clip representation for a linear classification head.
  • AST Model Architecture: SSAST replaces the original [CLS]-token representation with mean pooling because self-supervision is applied to individual patch representations.
  • Patch-level Masking: Clustered masking varies local and global context: larger C encourages global structure, smaller C encourages local structure, and random C exposes both.
  • AST Model Architecture: Non-overlapping patches are used during pretraining to prevent overlapped edges from becoming a shortcut, while fine-tuning and inference use overlap of 6.
  • MSPM: The training algorithm masks selected patch embeddings, encodes the sequence with positional information, computes both losses, averages them over masked patches, and updates the AST model.
  • MSPM: MSPM jointly applies discriminative and generative objectives to masked patches, using classification and reconstruction heads after Transformer encoding.

3 Experiments

Experiments show that MSPM self-supervision improves AST across audio and speech benchmarks, benefits from complementary objectives and combined pretraining data, and transfers across model sizes and patch shapes.

  • Pretraining setup: Unlabeled AudioSet and speech data are combined for self-supervised pretraining across audio and speech classification tasks.The framework uses audio data without labels and evaluates audio event, keyword, speaker, and emotion tasks.
  • Downstream performance: 60.9% average improvement over AST-Scratch is reported across six benchmarks, with SSAST reaching 0.310 mAP versus 0.148 mAP on AudioSet-20K.Improvement is reported as consistent across audio and speech benchmarks.
  • Pretraining ablations: 400 masked patches favor audio event classification, 250 favor speech tasks, while masking 100 patches produces the worst downstream performance.Overall performance for 250 and 400 masked patches is similar.
  • Pretraining ablations: Joint discriminative and generative objectives outperform either objective alone, indicating that the two MSPM tasks are complementary.Discriminative modeling performs better than generative modeling individually, while the joint objective performs best.
  • Pretraining data: Combining AudioSet-2M and Librispeech produces the best results, while MSPM improves AST even with limited AudioSet-20K pretraining.With AudioSet-20K used for both pretraining and fine-tuning, SSAST achieves 0.257 mAP and outperforms AST-Scratch.
  • Model size: MSPM consistently improves small, base, and larger AST architectures, and enables larger models to outperform smaller models after pretraining.From scratch, larger models do not always perform best on AS, KS1, and KS2.
  • Patch and frame designs: MSPM supports both patch-based and frame-based ASTs, reducing the frame-based advantage on speech tasks and improving patch-based models more strongly.Patch-based AST performs better on audio tasks, while frame-based AST remains stronger on speech tasks after MSPM.
  • Comparison with speech pretraining: SSAST matches or outperforms APC and wav2vec 1.0 on all three evaluated benchmarks.The comparison reports both fine-tuned and frozen settings for APC and wav2vec 1.0.

4 Related Work

Prior work established that pure Transformer architectures can outperform CNNs in vision and audio, while highlighting their greater data requirements and the limited development of patch-based self-supervision for audio.

  • Vision Transformers and Audio Spectrogram Transformers achieved strong performance without conventional non-degenerated convolutions.
  • Patch-based self-supervised learning had been extensively studied in vision, but self-supervised AST and patch-based audio self-supervision remained underexplored.

5 Conclusion

The paper introduces MSPM self-supervised pretraining for AST using both AudioSet and Librispeech, targeting generalization across audio and speech tasks. Experiments report broad gains, complementary objectives and datasets, and task-dependent patch-format advantages.

  • MSPM pretraining reduced AST’s need for labeled data and was evaluated across six downstream benchmarks spanning audio event classification, keyword spotting, speaker identification, and emotion recognition.
  • 60.9% average improvement was reported across downstream tasks, with SSAST matching or outperforming previous supervised pretrained models.
  • Joint generative and discriminative objectives outperformed either objective alone, while combining AudioSet and Librispeech outperformed single-domain pretraining.
  • After MSPM pretraining, patch-based AST performed better on audio tasks, whereas frame-based AST performed better on speech tasks.

Downstream Task Dataset and Evaluation Protocol Details

The paper provides additional details about the downstream datasets and the protocols used to evaluate the models.

  • The section describes downstream datasets and evaluation protocols used in the experiments.

AudioSet-20K

The evaluation includes AudioSet-20K for multi-label audio event classification and ESC-50 for single-label environmental audio classification.

  • AudioSet-20K: AudioSet-20K contains 20,785 audio clips and is evaluated on a disjoint AudioSet evaluation set using mean averaged precision.
  • ESC-50: ESC-50 contains 2,000 five-second recordings across 50 classes and is evaluated with standard five-fold cross-validation and accuracy.
  • ESC-50: ESC-50 has single-event clips and one-tenth the data volume of AudioSet-20K.

Speech Commands V2-35

Speech Commands V2-35 provides 105,829 one-second recordings across 35 common speech commands for keyword spotting. The model is fine-tuned on the training set, selected on validation data, and evaluated by test accuracy.

  • Speech Commands V2-35 contains 105,829 one-second recordings of 35 common speech commands.The dataset is split into 84,843 training, 9,981 validation, and 11,005 test samples.
  • The pretrained model is fine-tuned on the training set and selected using the validation set.
  • Keyword-spotting performance is reported as accuracy on the test set.

Speech Commands V1

Speech Commands V1 evaluates keyword spotting across keyword, silence, and unknown classes. The study uses the SUPERB framework and reports test-set accuracy.

  • Speech Commands V1 contains 10 keyword classes, one silence class, and one unknown class for false positives.
  • The keyword-spotting task uses the SUPERB evaluation framework.
  • Performance is reported as accuracy on the test set.

VoxCeleb 1

VoxCeleb 1 supports speaker identification using speech from 1,251 speakers, while the experimental setup also describes emotion recognition evaluation and AST implementation choices. Results are reported with task-specific accuracy measures.

  • VoxCeleb 1: VoxCeleb 1 contains 352 hours of speech from 1,251 speakers for speaker-identification classification.Training and testing use the same predefined speaker set, with test-set accuracy reported through SUPERB.
  • IEMOCAP: IEMOCAP provides about 12 hours of emotional speech from 10 speakers for emotion recognition.The evaluation retains four roughly balanced emotions, uses 5-fold cross-validation, and reports mean accuracy.
  • Representation: AST-Scratch and SSAST use mean-pooled patch representations, whereas supervised pretrained baselines use the [CLS] token representation.
  • Implementation: All AST models use default PyTorch weight initialization, matching the original AST paper.
  • Implementation: Experiments run with Python 3.7.4, PyTorch 1.9.0, CUDA 10.2, and four NVIDIA GTX Titan X-class GPUs.The hardware setup also includes Intel Xeon CPUs and 128GB memory.
Loading 2110.09784v2…