PaperlayerView exact revision on arXiv

Source-linked AI summary

BEATs: Audio Pre-Training with Acoustic Tokenizers

Sanyuan Chen, Yu Wu, Chengyi Wang, Shujie Liu, Daniel Tompkins, Zhuo Chen, Furu Wei

arXiv:2212.09058v1eess.AScs.AIcs.CLcs.LGcs.SD

TL;DR

Audio SSL models have largely relied on reconstruction despite the potential semantic advantages of discrete labels, while general audio lacks an obvious semantic tokenizer. BEATs iteratively trains an acoustic tokenizer and masked-label audio SSL model through mutual knowledge distillation, achieving SOTA results across six tasks, including 50.6 mAP on AudioSet-2M.

  • Problem

    State-of-the-art audio SSL models use reconstruction loss, while semantic-rich discrete labels are difficult to obtain for continuous, diverse general audio.

  • Method

    BEATs alternates masked discrete-label prediction for an audio SSL model with knowledge distillation from that model into the next acoustic tokenizer.

  • Results

    BEATs achieves superior performance across six audio and speech classification tasks, including 50.6 mAP on AudioSet-2M without external data.

  • Takeaways & Limitations

    The learned acoustic tokenizers produce discrete labels reported to be robust to disturbances and aligned with audio semantics.

Abstract

from arXiv · show

The massive growth of self-supervised learning (SSL) has been witnessed in language, vision, speech, and audio domains over the past few years. While discrete label prediction is widely adopted for other modalities, the state-of-the-art audio SSL models still employ reconstruction loss for pre-training. Compared with reconstruction loss, semantic-rich discrete label prediction encourages the SSL model to abstract the high-level audio semantics and discard the redundant details as in human perception. However, a semantic-rich acoustic tokenizer for general audio pre-training is usually not straightforward to obtain, due to the continuous property of audio and unavailable phoneme sequences like speech. To tackle this challenge, we propose BEATs, an iterative audio pre-training framework to learn Bidirectional Encoder representation from Audio Transformers, where an acoustic tokenizer and an audio SSL model are optimized by iterations. In the first iteration, we use random projection as the acoustic tokenizer to train an audio SSL model in a mask and label prediction manner. Then, we train an acoustic tokenizer for the next iteration by distilling the semantic knowledge from the pre-trained or fine-tuned audio SSL model. The iteration is repeated with the hope of mutual promotion of the acoustic tokenizer and audio SSL model. The experimental results demonstrate our acoustic tokenizers can generate discrete labels with rich audio semantics and our audio SSL models achieve state-of-the-art results across various audio classification benchmarks, even outperforming previous models that use more training data and model parameters significantly. Specifically, we set a new state-of-the-art mAP 50.6% on AudioSet-2M for audio-only models without using any external data, and 98.1% accuracy on ESC-50. The code and pre-trained models are available at https://aka.ms/beats.

1 Introduction

BEATs addresses the difficulty of applying discrete label prediction to general audio by iteratively co-optimizing an acoustic tokenizer and an audio SSL model. The resulting models achieve strong performance across audio and speech benchmarks.

  • State-of-the-art audio SSL models primarily use reconstruction loss, which may capture low-level features while neglecting high-level audio semantics.
  • Discrete label prediction is proposed as a more semantic-rich objective that encourages models to discard redundant details and improve modeling efficiency.
  • General audio lacks straightforward semantic tokenization because its signals are continuous, variable-duration, and diverse beyond speech.
  • BEATs alternates masked discrete-label prediction by the audio SSL model with knowledge distillation from that model into the next acoustic tokenizer.
  • 48.6 vs. 47.4 single-model and 50.6 vs. 49.6 ensemble performance are reported on AudioSet-2M, with BEATs also reducing ESC-50 error by 25% relative to prior SOTA.
  • The contributions include iterative audio pre-training, semantic-rich acoustic tokenizers, and SOTA results on audio and speech understanding benchmarks.

2 Related Work

Prior audio pre-training uses supervised, contrastive, reconstruction, or other self-supervised objectives, while BEATs explores masked discrete label prediction for audio. Its tokenizer is trained under supervision from the previous SSL model.

  • Supervised audio pre-training uses out-of-domain data such as ImageNet or in-domain audio data such as AudioSet.
  • Self-supervised audio pre-training commonly relies on contrastive learning or reconstruction objectives using large-scale unlabeled data.
  • Other approaches include temporal-distance estimation, patch permutation, and differentiable ranking for audio pre-training.
  • Unlike prior auto-encoding and ad-hoc clustering methods, BEATs trains an acoustic tokenizer under supervision from the previous SSL model.

3.1 Iterative Audio Pre-training

BEATs iteratively trains an acoustic tokenizer and an audio SSL model on unlabeled audio. Tokenizer-generated labels train masked prediction, while the converged SSL model supplies distillation targets for the next tokenizer.

  • Each iteration uses an acoustic tokenizer to generate discrete labels and trains the audio SSL model with masked discrete label prediction.
  • The converged audio SSL model becomes a teacher that trains a new acoustic tokenizer through knowledge distillation.
  • The input audio is converted into acoustic features, regular-grid patches, and a patch sequence X.
  • Tokenizer quantization produces patch-level labels as masked prediction targets, while the teacher produces output sequences as tokenizer distillation targets.
  • A pre-trained or fine-tuned SSL model can serve as teacher, with fine-tuning adding supervised semantic knowledge to the distillation process.

3.2 Acoustic Tokenizers

BEATs uses a random-projection tokenizer for cold-start labels, then replaces it with self-distilled tokenizers guided by prior SSL models. Quantization and distillation are designed to retain semantic information while reducing redundancy.

  • 3.2.1 Cold Start: Random-Projection Tokenizer: The first BEATs iteration uses a frozen random-projection tokenizer to cluster continuous acoustic features into discrete labels.
  • 3.2.1 Cold Start: Random-Projection Tokenizer: Random projection maps each patch through a random linear layer, then assigns the nearest codebook vector's index as its discrete label.
  • 3.2.2 Iteration: Self-Distilled Tokenizer: From the second iteration, a previous pre-trained or fine-tuned SSL model teaches a self-distilled tokenizer to generate refined patch-level labels.
  • 3.2.2 Iteration: Self-Distilled Tokenizer: The self-distilled tokenizer encodes patches, quantizes them with learnable codebook embeddings, and trains an estimator to predict teacher outputs.
  • 3.2.2 Iteration: Self-Distilled Tokenizer: Knowledge distillation optimizes tokenized labels toward semantic-rich teacher knowledge and away from redundant input information.
  • 3.2.2 Iteration: Self-Distilled Tokenizer: Straight-through gradients address vector quantization's non-differentiability by copying gradients from quantized vectors to encoded vectors.
  • 3.2.2 Iteration: Self-Distilled Tokenizer: The tokenizer objective combines cosine similarity for estimator outputs with mean squared error for encoded representations, using stop-gradient and exponential moving averages.

3.3 Audio SSL Model

BEATs uses a ViT-based audio SSL model that predicts masked patch-level discrete acoustic labels rather than reconstructing acoustic features. During fine-tuning, the label predictor is removed and a task-specific classifier uses the encoder representations for downstream classification.

  • Model architecture: The audio SSL backbone uses a ViT with a linear projection layer followed by stacked Transformer encoder layers.The Transformer includes convolution-based relative position embeddings, gated relative position bias, and DeepNorm for stable pre-training.
  • Pre-training: Masked Audio Modeling predicts patch-level discrete labels generated by acoustic tokenizers with a Transformer-based label predictor.This replaces acoustic-feature reconstruction as the pre-training target.
  • Pre-training: 75% of input patches are randomly masked, and only the unmasked patches are fed into the ViT encoder before label prediction.The unmasked representations are combined with masked patch features to predict the full discrete label sequence.
  • Pre-training: The MAM objective is cross entropy over the correct acoustic labels at masked positions conditioned on the unmasked patch sequence.Feeding only non-masked patches into the encoder significantly speeds training while slightly improving downstream performance.
  • Fine-tuning: During fine-tuning, the label predictor is discarded and a task-specific linear classifier is attached to the ViT encoder.The full patch sequence is encoded, representations are mean-pooled, and classification probabilities are computed with softmax.

4 Experiment

BEATS is evaluated on six audio and speech classification tasks using iterative tokenizer and audio SSL-model training, with self-distilled tokenizers refining semantic targets. The models achieve strong results across tasks, including new or improved state-of-the-art performance on AudioSet and ESC-50.

  • Evaluation setup: BEATS is evaluated on six downstream tasks: AS-2M, AS-20K, ESC-50, KS1, KS2, and ER.The evaluations cover three audio classification tasks and three speech classification tasks.
  • Model and tokenizer training: 90M-parameter BEATS models are pretrained on AS-2M for three iterations, progressing from random-projection labels to self-distilled tokenizer targets.BEATSiter1 uses random-projection labels; later iterations use tokenizers guided by pretrained or fine-tuned BEATS teachers.
  • Single-model results: BEATS achieves the best performance across all six audio and speech classification tasks.BEATSiter3+ reaches 48.6 versus 47.4 mAP on AS-2M and 38.9 versus 37.6 on AS-20K, with 90M versus 304M parameters.
  • Single-model results: BEATSiter1 outperforms previous works on five of six tasks, while BEATSiter2 further improves performance with refined self-distilled labels.The reported gains support discrete label prediction and semantic refinement through iterative tokenizer training.
  • Iterative refinement: BEATSiter3 performs similarly to BEATSiter2, whereas supervised-teacher BEATSiter3+ significantly improves AS-2M and AS-20K results.The authors describe fast convergence in a few iterations and gains from using fine-tuned BEATSiter2 as the teacher.
  • Tokenizer comparison: The self-distilled tokenizer is insensitive to different self-supervised teachers but performs best when guided by an AS-2M fine-tuned BEATSiter2 teacher.The best tokenizer performance is reported across all three audio classification tasks.
  • Pre-training target visualization: BEATS pre-training targets are more robust to waveform disturbances and capture semantic content more effectively than reconstruction-based targets.Self-supervised teachers help cluster semantically similar samples and remove background reverberation and noise; supervised teachers capture high-level semantics despite redundant low-level details.
  • Ensemble results: BEATS ensembles achieve 50.6 mAP on AS-2M without external supervised data.The ten-model ensemble further improves the five-model ensemble and sets the reported state-of-the-art result.

5 Conclusion

BEATS introduces an iterative self-supervised framework that uses self-distilled acoustic tokenizers for discrete-label audio pre-training. It achieves superior performance across six classification tasks and sets new state-of-the-art results on AudioSet-2M and ESC-50, while future work targets larger models, more data, and multimodality.

  • BEATS converts continuous audio into discrete labels with a self-distilled tokenizer, enabling mask and discrete-label prediction instead of reconstruction loss.
  • BEATS achieves superior performance across six audio and speech classification tasks, including state-of-the-art results on AudioSet-2M and ESC-50.
  • Visualization indicates that BEATS pre-training targets are more robust to disturbances and more semantically aligned than reconstruction-based targets.
  • Future work will scale model size and pre-training data and explore combining audio with vision and language.

A.1 Hyperparamter Settings

This section specifies the hyperparameters used for BEATS tokenizer training, audio SSL pre-training, and fine-tuning. The settings are adapted from previous works, with per-epoch class balancing described in the table caption.

  • The listed settings cover three stages: acoustic tokenizer training, audio SSL model pre-training, and downstream fine-tuning.
  • Table 4 lists hyperparameters for BEATS acoustic tokenizer training, audio SSL model pre-training, and fine-tuning.The settings are adapted from previous works.
  • Each class is balanced to 50% of the unknown class size for each training epoch.
Loading 2212.09058v1…