Source-linked AI summary
AST: Audio Spectrogram Transformer
Yuan Gong, Yu-An Chung, James Glass
TL;DR
Audio classification has largely relied on CNNs, leaving open whether attention alone is sufficient. The paper introduces AST, a convolution-free Transformer applied directly to spectrograms, and reports state-of-the-art results across AudioSet, ESC-50, and Speech Commands V2.
Problem
The paper asks whether CNNs are necessary for audio classification or whether purely attention-based networks can perform well.
Method
AST applies a convolution-free Transformer directly to audio spectrograms, with ImageNet-pretrained ViT knowledge transferred to improve performance.
Results
AST outperforms state-of-the-art systems across AudioSet, ESC-50, and Speech Commands, including 0.485 mAP on AudioSet, 95.6% accuracy on ESC-50, and 98.1% accuracy on Speech Commands V2.
Takeaways & Limitations
The paper concludes that CNNs are not indispensable for audio classification and presents AST as a simple attention-based alternative with superior performance.
Takeaways & Limitations
AST remains convolution-like in some components: the patch embedding can be viewed as a large-kernel convolution and Transformer projections as 1×1 convolutions.
Abstract
from arXiv · showhide
In the past decade, convolutional neural networks (CNNs) have been widely adopted as the main building block for end-to-end audio classification models, which aim to learn a direct mapping from audio spectrograms to corresponding labels. To better capture long-range global context, a recent trend is to add a self-attention mechanism on top of the CNN, forming a CNN-attention hybrid model. However, it is unclear whether the reliance on a CNN is necessary, and if neural networks purely based on attention are sufficient to obtain good performance in audio classification. In this paper, we answer the question by introducing the Audio Spectrogram Transformer (AST), the first convolution-free, purely attention-based model for audio classification. We evaluate AST on various audio classification benchmarks, where it achieves new state-of-the-art results of 0.485 mAP on AudioSet, 95.6% accuracy on ESC-50, and 98.1% accuracy on Speech Commands V2.
1. Introduction
AST addresses whether CNNs are necessary for audio classification by applying attention directly to spectrograms. It is presented as a convolution-free model that differs from CNN-based and CNN-attention approaches.
- Motivation: CNNs have dominated end-to-end audio classification, while self-attention has been added to CNNs to capture long-range global context.These models map audio spectrograms directly to labels and use CNN inductive biases such as spatial locality and translation equivariance.
- Contribution: AST is a convolution-free, purely attention-based model applied directly to audio spectrograms.Its attention mechanism can capture long-range global context even in the lowest layers.
- Contribution: AST uses a common architecture across tasks whose input lengths vary from 1 sec. to 10 sec.The paper contrasts this with CNN-based models, which typically require architecture tuning for different tasks.
- Contribution: AST is described as simpler, smaller, and faster-converging than CNN-attention hybrid models.The paper also identifies AST as, to the authors’ knowledge, the first purely attention-based audio classification model.
- Related Work: AST differs from related audio models by removing convolution while retaining Transformer-based attention mechanisms.The paper relates AST most closely to the Vision Transformer but emphasizes AST’s variable-length audio capability and cross-modal transfer from ImageNet-pretrained ViT.
2. Audio Spectrogram Transformer
AST converts audio spectrograms into patch embeddings with positional information and processes them using a standard Transformer encoder. Its design is called convolution-free while acknowledging limited convolution-like operations in the embedding and projection layers.
- 2.1. Model Architecture: AST converts audio into a 128×100t log Mel spectrogram and splits it into overlapping 16×16 patches.Each patch is flattened and linearly projected into a 768-dimensional embedding.
- 2.1. Model Architecture: Trainable positional embeddings are added to patch embeddings so the Transformer can capture the spectrogram’s spatial structure.The patch sequence itself is not in temporal order, and Transformers do not otherwise capture input order information.
- 2.1. Model Architecture: A prepended [CLS] token is processed by an unmodified Transformer encoder for classification.The setup uses only encoder layers and retains the standard Transformer architecture.
- 2.1. Model Architecture: AST is termed convolution-free despite a patch embedding layer and Transformer projections that can be viewed as convolution-like operations.The distinction is from conventional CNNs with multiple layers and small kernels and strides.
- 2.2. ImageNet Pretraining: ImageNet-to-AST transfer learning addresses the Transformer’s greater data requirements relative to CNNs.The paper motivates this strategy because audio datasets typically lack the very large sample sizes associated with Transformer advantages in vision.
- 2.2. ImageNet Pretraining: ViT weights are adapted to AST by averaging the three input-channel weights and processing positional embeddings for variable-length spectrograms.AST and ViT share a standard Transformer, patch size, and embedding size, but their input channels and shapes differ.
3. Experiments
AST is evaluated on AudioSet, ESC-50, and Speech Commands V2 through primary benchmarks and ablations covering pretraining, positional embeddings, patch overlap, and patch geometry. Across these experiments, AST outperforms prior systems, including with relatively small training sets.
- Experimental scope: AST experiments cover AudioSet, ESC-50, and Speech Commands V2, including benchmark comparisons and ablations of major design choices.The ablations examine ImageNet pretraining, positional embedding adaptation, patch overlap, and patch shape and size.
- AudioSet results: 0.485 mAP is achieved by the full AudioSet Ensemble-M model, compared with 0.459±0.000 for the best weight-averaged single model.Ensemble-M combines models trained with different random seeds and patch split strategies.
- AudioSet results: 0.378 mAP is achieved by the balanced AudioSet Ensemble-M model, while the single model reaches 0.347±0.001.These results use about 1% of the full AudioSet training set, and all reported AST variants outperform the previous best system.
- Ablation study: ImageNet pretraining noticeably improves AST performance on both balanced and full AudioSet, with a larger benefit when less in-domain audio data is available.The ablation reports that pretraining can greatly reduce AST’s demand for in-domain audio data.
- Ablation study: Patch-overlap performance improves as overlap increases, but longer Transformer sequences raise computational overhead quadratically.Even without overlap, AST still outperforms the previous best system.
- ESC-50 and Speech Commands results: 95.6±0.4% accuracy is achieved by AST-P on ESC-50, while AST-S reaches 88.7±0.7%, and both outperform same-setting state-of-the-art models.AST-P uses ImageNet and AudioSet pretraining; AST-S uses ImageNet pretraining only.
4. Conclusions
AST shows that CNNs are not indispensable for audio classification: a convolution-free, purely attention-based model can combine a simple architecture with superior performance.
- AST is a convolution-free, purely attention-based model for audio classification.