Source-linked AI summary
Efficient Training of Audio Transformers with Patchout
Khaled Koutini, Jan Schlüter, Hamid Eghbal-zadeh, Gerhard Widmer
TL;DR
The paper addresses the quadratic computation and memory costs that limit transformer training on audio spectrograms. It proposes Patchout alongside disentangled positional encodings and evaluates these choices across audio tasks, achieving state-of-the-art performance while enabling efficient single-GPU training and faster training than CNNs in the reported comparisons.
Problem
Transformer attention has O(n^2) computation and memory complexity with sequence length, creating efficiency challenges for longer audio clips and overlapping spectrogram patches.
Method
The paper proposes Patchout to remove spectrogram patches during training, together with separate time and frequency positional encodings and additional complexity-reduction methods.
Results
The proposed models achieve state-of-the-art performance on Audioset and downstream tasks, while PaSST-S-L and PaSST-S-N outperform CNNs in training speed with comparable memory requirements.
Takeaways & Limitations
Patchout reduces training compute and memory requirements while functioning as a regularizer, supporting efficient transformer training for audio applications.
Takeaways & Limitations
The experiments use ImageNet-pretrained models throughout, making ImageNet pretraining an explicit scope condition of the reported evaluation.
Abstract
from arXiv · showhide
The great success of transformer-based models in natural language processing (NLP) has led to various attempts at adapting these architectures to other domains such as vision and audio. Recent work has shown that transformers can outperform Convolutional Neural Networks (CNNs) on vision and audio tasks. However, one of the main shortcomings of transformer models, compared to the well-established CNNs, is the computational complexity. In transformers, the compute and memory complexity is known to grow quadratically with the input length. Therefore, there has been extensive work on optimizing transformers, but often at the cost of degrading predictive performance. In this work, we propose a novel method to optimize and regularize transformers on audio spectrograms. Our proposed models achieve a new state-of-the-art performance on Audioset and can be trained on a single consumer-grade GPU. Furthermore, we propose a transformer model that outperforms CNNs in terms of both performance and training speed. Source code: https://github.com/kkoutini/PaSST
1. Introduction
Transformers offer powerful sequence modeling for audio but incur quadratic compute and memory costs as spectrogram sequence length grows. This work introduces Patchout and related design choices to reduce training complexity while preserving or improving audio-task performance.
- Transformers compute pairwise interactions across sequence items, making attention complexity O(n^2) in both computation and memory.This quadratic cost becomes problematic for longer audio clips or overlapping patches.
- The paper introduces Patchout to reduce transformer computation and memory complexity while also regularizing audio models.Patchout is presented as a simple method for training transformers on audio spectrograms.
- Disentangled time and frequency positional encodings support inference on variable-length audio snippets without fine-tuning or positional-encoding interpolation.For shorter clips, the time positional encoding can be cropped while the frequency encoding remains unchanged.
- The study evaluates additional complexity-reduction methods on Audioset and domain-specific downstream audio tasks.The investigation examines how these methods affect performance across general-purpose and downstream settings.
- The proposed models achieve state-of-the-art performance on several audio tagging and classification tasks using a single consumer GPU in relatively short training times.When complexity-reduction methods are combined, the models also outperform CNNs in training speed, memory requirements, and generalization.
2. The Patchout faSt Spectrogram Transformer (PaSST)
PaSST adapts vision-transformer processing to audio spectrograms by combining patch extraction, separate time and frequency positional encodings, and Patchout. These choices reduce transformer complexity while supporting variable-length audio and several model variants.
- Architecture: The PaSST pipeline projects spectrogram patches, adds time and frequency positional encodings, applies Patchout, and classifies transformed C and D tokens.The distillation token D is used only in DeiT-based models.
- Positional Encoding: Disentangled time and frequency encodings allow shorter-audio inference and fine-tuning by cropping time-position parameters without changing frequency encodings.This avoids fine-tuning or interpolating positional encodings for variable-length inputs.
- Complexity Analysis: Multi-head attention has O(n^2) compute and memory complexity because it forms an n × n attention matrix.The remaining network operations scale linearly with sequence length, making sequence reduction especially impactful.
- Patchout: Patchout drops parts of the spectrogram patch sequence during training, reducing computation while encouraging classification from incomplete inputs.The method is motivated by reducing sequence length and the expectation that audio events are distributed across time and frequency.
- Patchout: Unstructured Patchout randomly removes patches regardless of position, producing models called PaSST-U.Structured Patchout instead removes complete rows or columns corresponding to randomly selected frequency bins or time frames, producing PaSST-S.
- Complexity Reduction: Reducing patch overlap shortens the input sequence but can degrade Audioset performance; Patchout also operates without overlap in PaSST-N.The paper additionally reduces transformer depth by removing alternating self-attention blocks, yielding PaSST-L with 7 blocks and 50M parameters versus 87M.
3. Experiment Setup
The experiments train PaSST on Audioset and fine-tune it for multiple audio tasks, using ImageNet-pretrained models and audio-specific augmentation. The setup compares Patchout variants, overlap choices, and reduced transformer depth.
- Datasets and Tasks: Audioset training uses around 2 million YouTube clips labeled across 527 classes, followed by fine-tuning for instrument, environmental, and acoustic-scene tasks.Audioset is described as the largest publicly available audio dataset.
- Feature Extraction: The models use mono 32 kHz audio with 128 Mel bands extracted from 25 ms windows and 10 ms hops.Audioset training data is balanced using importance sampling with weights proportional to inverse label frequency.
- Pretraining: All experiments use ImageNet-pretrained models, with DeiT B↑384 as the base model.A reduced-depth ViT-hug224 variant is also reported as having comparable performance but is not explored further.
- Augmentation: Augmentation combines waveform and spectrogram Mix-Up, SpecAugment masking, random time rolling, and random gain changes of ±7 dB.SpecAugment masks up to 48 frequency bins and 192 time frames.
4. Results
PaSST achieves state-of-the-art Audioset performance while substantially reducing training time and GPU memory, and transfers effectively to downstream audio tasks. Structured Patchout and eliminating patch overlap provide efficient fine-tuning alternatives to reducing transformer depth.
- Audio Tagging on Audioset: PaSST achieves new state-of-the-art performance on Audioset while training on a single consumer GPU.PaSST reaches state-of-the-art performance in 50 hours on a single Nvidia RTX 2080ti.
- Audio Tagging on Audioset: 4× faster training and less than 25% of the required GPU memory are achieved using Patchout.Patchout improves transformer performance while increasing training speed approximately four times and reducing required GPU memory to less than 25%.
- Audio Tagging on Audioset: PaSST-S-L and PaSST-S-N outperform CNNs while maintaining higher training throughput and similar GPU memory requirements.PaSST-S-L and PaSST-S-N reach .459 and .466 mAP, respectively, in approximately 25 hours on a single GPU.
- Fine-tuning and Transfer to Downstream Tasks: Fine-tuned PaSST outperforms state-of-the-art CNNs on all downstream tasks, with Patchout often improving generalization and substantially accelerating training.Structured Patchout combined with no patch overlap performs better than reducing transformer depth and enables faster fine-tuning.
- Fine-tuning and Transfer to Downstream Tasks: PaSST-S-N reaches state-of-the-art OpenMIC performance in under 30 minutes, while PaSST variants fine-tune on ESC50 in under 5 minutes.OpenMIC uses mean average precision, and ESC50 reports accuracy averaged over five official folds.
5. Conclusion
The paper presents Patchout for efficient transformer training on audio spectrograms, achieving state-of-the-art results on Audioset and downstream tasks. PaSST variants outperform CNNs with faster training and comparable memory, while pretrained models support resource-efficient fine-tuning.
- Patchout enables efficient transformer training on audio spectrograms while achieving state-of-the-art performance on Audioset and downstream tasks.
- PaSST-S-L and PaSST-S-N outperform CNNs with faster training and comparable memory requirements.
- Pretrained PaSST models can be fine-tuned on downstream audio tasks with little computational resources and additional training time.