Source-linked AI summary
ElasticAST: An Audio Spectrogram Transformer for All Length and Resolutions
Jiu Feng, Mehmet Hamza Erol, Joon Son Chung, Arda Senocak
TL;DR
ASTs use fixed-size inputs, limiting their fit to audio of varying lengths and resolutions. ElasticAST packs variable-length audio during training and applies sample-specific attention and pooling. It supports inference across lengths and resolutions with performance comparable to fixed-input ASTs, and reports better results on variable-length datasets.
Problem
ASTs use fixed-size inputs, while audio datasets and inference requirements can involve varying lengths or temporal resolutions.
Source: §1 · p. 1Method
ElasticAST trains on native-length audio or randomly mixed resolutions by packing samples into sequences and restricting attention and pooling to each sample.
Source: §1 · p. 2, §2.2 · pp. 2–3Results
ElasticAST operates across lengths and resolutions at inference without significant performance degradation, matches fixed-input ASTs, and surpasses them on variable-length datasets.
Source: §1 · p. 2Takeaways & Limitations
ElasticAST offers one model adaptable to different computational budgets without retraining.
Source: §4 · p. 4
Abstract
from arXiv · showhide
Transformers have rapidly overtaken CNN-based architectures as the new standard in audio classification. Transformer-based models, such as the Audio Spectrogram Transformers (AST), also inherit the fixed-size input paradigm from CNNs. However, this leads to performance degradation for ASTs in the inference when input lengths vary from the training. This paper introduces an approach that enables the use of variable-length audio inputs with AST models during both training and inference. By employing sequence packing, our method ElasticAST, accommodates any audio length during training, thereby offering flexibility across all lengths and resolutions at the inference. This flexibility allows ElasticAST to maintain evaluation capabilities at various lengths or resolutions and achieve similar performance to standard ASTs trained at specific lengths or resolutions. Moreover, experiments demonstrate ElasticAST's better performance when trained and evaluated on native-length audio datasets.
1. Introduction
Transformers have displaced CNNs in audio, but ASTs retain fixed-size inputs that constrain variable-length audio. ElasticAST trains and infers across lengths and resolutions, with performance comparable to fixed-input ASTs and better on variable-length datasets.
1. Introduction
- Variable-length recordings make fixed-size AST inputs problematic: trimming or padding can discard or contaminate information, and differing inference lengths may require separate models.The paper identifies varied computation budgets and memory consumption as cases where sequence-length requirements differ.Source: §1 · p. 1
- Prior AST flexibility work changes patch sizes, while mixed-resolution training targets efficiency; ElasticAST instead mixes resolutions and native lengths without curriculum.The paper also draws inspiration from NaViT’s flexibility across image resolutions.Source: §1 · pp. 1–2
- ElasticAST randomly mixes audio resolutions or native lengths during training, packs samples into sequences, and restricts attention and pooling to each sample.It uses minimal changes to the standard AST and does not use a curriculum for mixing resolutions.Source: §1 · p. 2
- One ElasticAST handles all lengths and resolutions at inference without significant performance degradation, matching fixed-input ASTs and surpassing them on variable-length datasets.The reported gains include VoxCeleb and Epic-Sounds.Source: §1 · p. 2
2. Approach
AST converts spectrograms into patch embeddings, prepends a class token, and uses the encoder’s class-token output as the audio representation. ElasticAST packs variable-length samples into token rows, prevents cross-sample attention, and pools representations separately for each sample.
2.1. Preliminaries
- AST splits each spectrogram into patches, linearly embeds them, adds positional embeddings, and uses the encoder output for a prepended class token as the audio representation.The class-token representation supports downstream tasks such as classification.Source: §2.1 · p. 2
2.2. Architectural changes
- Sequence packing fills token rows sequentially with samples up to a preset limit, deferring samples that would exceed it and minimally padding rows to a common length.The default row token limit is 2048; samples are processed in their original order.Source: §2.2 · p. 2
- Masked self-attention and masked attention pooling keep packed samples separate, then produce a representation for each sample for the classifier.Attention is restricted within each sample, and pooling extracts representations only from that sample’s tokens.Source: §2.2 · p. 3
3. Experiments
ElasticAST handles variable-length audio through training and inference, while standard ASTs trim or pad inputs and are less flexible. Across varied lengths and resolutions, ElasticAST retains performance, and its token usage analysis finds less padding and no training-time cutting.
3.1. Datasets and Evaluation Metrics
- The experiments use AudioSet, VGGSound, VoxCeleb, and Epic-Sounds, with mAP for AudioSet and Top-1 accuracy for the other datasets.VoxCeleb and Epic-Sounds are capped at 30 seconds for the experiments.Source: §3.1 · p. 3
3.2. Implementation Details
- ElasticAST uses the same settings as standard ASTs, with separate frequency and time positional embeddings to accommodate varying audio lengths.The standard configuration uses batch size 12, patch size 16, and a 10 ms frame shift by default.Source: §3.2 · p. 3
3.3. Results of Various Length Training
- ElasticAST performs well across native audio lengths, while standard AST performance collapses when inference length differs from training.On native-length audio, ElasticAST can surpass standard ASTs by using full semantic content without trimming or padding.Source: §3.3 · pp. 3–4, Figure 4 · p. 3
3.4. Results of Various Temporal Resolutions Training
- ElasticAST flexibly handles varied temporal resolutions across VGGSound and AudioSet, while standard AST performance declines at resolutions different from training.ElasticAST generally matches or surpasses standard ASTs, including at resolutions for which they were trained.Source: §3.4 · p. 4, Figure 5 · p. 4
3.5. Ablation Study
- Average pooling, using kernel and stride 1×C, reduces mel-spectrogram temporal dimension by C; ElasticAST remains flexible across tested temporal lengths.The VGGSound ablation tests integer compression rates C = {1, 2, 3, 4}.Source: §3.5 · p. 4, Table 1 · p. 4
3.6. Analysis on Token Usage
- Standard AST uses 31.1% non-informative padding tokens and cuts 15.2% of training tokens; ElasticAST minimizes packing padding and applies no cutting.As batch size increases, packing uses fewer padding tokens.Source: §3.6 · p. 4, Table 1 · p. 4
4. Conclusion
ElasticAST uses mixed-length training and minimal architectural changes to support AST training and inference across audio lengths. It offers flexibility without performance loss across lengths, with one model for audio lengths and resolutions.
4. Conclusion
- ElasticAST supports AST training and inference with varied audio lengths using mixed-length training and minimal architectural changes.The conclusion describes one model for different audio lengths and resolutions, adaptable to computational budgets without retraining.Source: §4 · p. 4