PaperlayerView exact revision on arXiv

Source-linked AI summary

Masked Spectrogram Prediction For Self-Supervised Audio Pre-Training

Dading Chong, Helin Wang, Peilin Zhou, Qingcheng Zeng

arXiv:2204.12768v1cs.SDeess.AS

TL;DR

Limited audio data leaves transformer-based audio models dependent on cross-domain initialization, while prior audio self-supervised methods have underperformed downstream. MaskSpec masks random spectrogram patches and reconstructs them with an encoder-decoder using unlabeled AudioSet. The method outperforms prior pretraining approaches and supervised methods across multiple downstream datasets.

  • Problem

    Limited audio-domain data constrains transformer pretraining, while cross-domain initialization and existing audio self-supervised methods have important performance or transfer limitations.

    Source: §1 · p. 1

  • Method

    MaskSpec randomly masks input spectrogram patches and reconstructs the masked regions with an encoder-decoder during self-supervised pretraining.

    Source: §1 · p. 1, §2 · p. 2, §2.1 · p. 2

  • Results

    MaskSpec outperforms previous pre-trained models and supervised methods across multiple downstream datasets.

    Source: §1 · pp. 1–2, §3.3 · p. 4

  • Takeaways & Limitations

    MaskSpec learns transferable time-frequency representations from unlabeled audio without transferring weights from other domains.

    Source: §1 · p. 2, §4 · p. 4

Abstract

from arXiv · show

Transformer-based models attain excellent results and generalize well when trained on sufficient amounts of data. However, constrained by the limited data available in the audio domain, most transformer-based models for audio tasks are finetuned from pre-trained models in other domains (e.g. image), which has a notable gap with the audio domain. Other methods explore the self-supervised learning approaches directly in the audio domain but currently do not perform well in the downstream tasks. In this paper, we present a novel self-supervised learning method for transformer-based audio models, called masked spectrogram prediction (MaskSpec), to learn powerful audio representations from unlabeled audio data (AudioSet used in this paper). Our method masks random patches of the input spectrogram and reconstructs the masked regions with an encoder-decoder architecture. Without using extra model weights or supervision, experimental results on multiple downstream datasets demonstrate MaskSpec achieves a significant performance gain against the supervised methods and outperforms the previous pre-trained models. In particular, our best model reaches the performance of 0.471 (mAP) on AudioSet, 0.854 (mAP) on OpenMIC2018, 0.982 (accuracy) on ESC-50, 0.976 (accuracy) on SCV2, and 0.823 (accuracy) on DCASE2019 Task1A respectively.

1. Introduction

Audio transformer pretraining is constrained by limited labeled audio data and uncertainty about cross-domain transfer, while existing audio self-supervised methods underperform. MaskSpec addresses this gap by reconstructing randomly masked spectrogram patches and demonstrates strong downstream effectiveness without transferred weights.

  • Limited audio data constrains pretraining transformer-based audio models despite their strong performance when sufficient data are available.Source: §1 · p. 1
  • Cross-domain initialization from image models performs well, but modality mismatches make its effectiveness and transferability unclear [11] [12].Source: §1 · p. 1
  • Audio self-supervised methods reduce dependence on labeled data but had not matched the performance of transferred models.Source: §1 · p. 1
  • MaskSpec randomly masks spectrogram patches, removes them from the encoder input, and reconstructs their information and positions from surviving patches.Source: §1 · p. 1, Figure 1 · p. 2
  • MaskSpec is evaluated across five downstream tasks and is reported to outperform both from-scratch self-supervised and cross-domain transfer methods.Source: §1 · pp. 1–2

2. Masked Spectrogram Prediction

MaskSpec pretrains an audio transformer by masking spectrogram patches and reconstructing them with an asymmetric encoder-decoder. The framework removes masked patches from the encoder, restores their positions for decoding, and optimizes reconstruction against the original patches.

2. Masked Spectrogram Prediction

  • MaskSpec uses spectrograms rather than raw waveforms and reconstructs masked patches with an asymmetric encoder-decoder architecture.The paper motivates spectrograms by their acoustic information, image-like structure, direct transformer compatibility, and lower computational cost than waveform inputs requiring extra convolutional layers.Source: §2 · p. 2

2.1. Masking strategy

  • Random masking splits a spectrogram into patches, masks a preset ratio α, removes masked patches from the encoder, and preserves their positions for reconstruction.The masking ratio α ranges from 0.05 to 0.95 in the experiments, while the method directly removes masked patches for efficiency.Source: §2.1 · p. 2

2.2. Encoder

  • The encoder follows the PaSST architecture, with MaskSpec, MaskSpec-Small, and MaskSpec-Tiny providing three model scales.The full MaskSpec uses 12 transformer blocks, 12 attention heads, 768-dimensional embeddings, and feed-forward layers of size 2048.Source: §2.2 · p. 2

2.3. Decoder

  • The lightweight decoder inserts shared learnable vectors into masked positions, reassembles all patches with positional information, and predicts the masked patch contents.It uses 8 transformer layers and a final linear projection, and is used only during pretraining.Source: §2.3 · pp. 2–3, Figure 2 · p. 3

2.4. Implementation of framework

  • Training minimizes mean squared error between reconstructed masked patches and their original values after encoding surviving patches and decoding the completed sequence.The framework adds position information, inserts a learnable vector at each masked location, and maps decoder outputs back to the original patch dimensionality.Source: §2.4 · p. 3

3. Experiments

Experiments pre-train MaskSpec on unlabeled AudioSet, finetune it across diverse downstream tasks, and compare it with prior supervised and self-supervised methods. MaskSpec improves over random initialization and another self-supervised method, approaches cross-modal transfer on AudioSet, and performs strongly across downstream tasks.

3.1. Pre-training

  • MaskSpec pre-training uses unlabeled AudioSet, followed by finetuning on audio tagging, environmental sound, acoustic scene, music-instrument, and speech-command tasks.AudioSet contains over 2 million clips, and labels are not used during pre-training.Source: §3.1 · p. 3, §3.2 · pp. 3–4

3.3. Results and discussions

  • MaskSpec outperforms another self-supervised method and achieves results comparable to cross-modal transfer methods on full-data AudioSet finetuning.The comparison uses the same network structures where applicable and reports mAP against random initialization and prior pre-training methods.Source: Table 1 · p. 3, §3.3 · p. 4
  • Effective self-supervised pre-training works with masking ratios from 15% to 85%, with the best AudioSet result at a 75% masking ratio.The experiment varies the masking ratio α during AudioSet encoder finetuning.Source: §3.3 · p. 4
  • MaskSpec shows stronger generalization than SSAST across downstream tasks except SCV2, where SSAST performs slightly better after using extra Librispeech pre-training data.MaskSpec performs worse than AST on AudioSet-20K, which uses extra image data, while AudioSet finetuning before downstream transfer improves all tasks.Source: §3.3 · p. 4, Table 2 · p. 4
  • MaskSpec beats the compared supervised methods without extra data, indicating robustness and generalization across the evaluated downstream tasks.The experiments cover audio tagging, environment sound classification, acoustic scene classification, polyphonic music instrument recognition, and speech command recognition.Source: §3.3 · p. 4, §3.2 · p. 4

4. Conclusions

The paper presents MaskSpec, which reconstructs masked spectrogram patches with an asymmetric encoder-decoder to learn time-frequency representations from unlabeled audio. Results on AudioSet and downstream tasks show transfer ability, while future work targets larger unlabeled-audio training and more efficient finetuning.

4. Conclusions

  • MaskSpec reconstructs masked spectrogram patches with an asymmetric encoder-decoder structure to learn representations from unlabeled audio.The paper identifies stronger-capacity training with more unlabeled audio and more efficient finetuning as future directions.Source: §4 · p. 4

Related papers

Loading 2204.12768v1…