Source-linked AI summary
HTS-AT: A Hierarchical Token-Semantic Audio Transformer for Sound Classification and Detection
Ke Chen, Xingjian Du, Bilei Zhu, Zejun Ma, Taylor Berg-Kirkpatrick, Shlomo Dubnov
TL;DR
Existing audio transformers require large GPU memories and long training times, while class-token outputs limit event localization. HTS-AT introduces hierarchical processing with window attention and token-semantic mapping, achieving or matching state-of-the-art results across several datasets with substantially lower parameter and training-time requirements than AST.
Problem
Existing audio transformers require large GPU memories and long training times, and relying on pretrained vision models limits their scalability in audio tasks.
Method
HTS-AT combines a hierarchical transformer with window attention and a token-semantic module that maps final outputs into class featuremaps.
Results
HTS-AT achieves or equals state-of-the-art results on AudioSet, ESC-50, and Speech Command V2, while enabling event localization from weakly labeled data.
Takeaways & Limitations
HTS-AT is reported as a high-performance, high-scalability, lightweight audio transformer for classification and event localization.
Abstract
from arXiv · showhide
Audio classification is an important task of mapping audio samples into their corresponding labels. Recently, the transformer model with self-attention mechanisms has been adopted in this field. However, existing audio transformers require large GPU memories and long training time, meanwhile relying on pretrained vision models to achieve high performance, which limits the model's scalability in audio tasks. To combat these problems, we introduce HTS-AT: an audio transformer with a hierarchical structure to reduce the model size and training time. It is further combined with a token-semantic module to map final outputs into class featuremaps, thus enabling the model for the audio event detection (i.e. localization in time). We evaluate HTS-AT on three datasets of audio classification where it achieves new state-of-the-art (SOTA) results on AudioSet and ESC-50, and equals the SOTA on Speech Command V2. It also achieves better performance in event localization than the previous CNN-based models. Moreover, HTS-AT requires only 35% model parameters and 15% training time of the previous audio transformer. These results demonstrate the high performance and high efficiency of HTS-AT.
1. INTRODUCTION
HTS-AT addresses audio transformers’ high computational cost and limited event-localization capability with a hierarchical architecture and token-semantic module. It achieves strong benchmark performance while reducing model size and training time.
- Motivation: Audio transformers such as AST require substantial GPU memory and training time, and depend on vision pretraining for high performance.AST reportedly takes about one week to train on full AudioSet using four 12GB GPUs; without pretraining, it reaches mAP=0.366 on AudioSet.
- Proposed approach: HTS-AT combines a hierarchical audio transformer with a token-semantic module for audio classification.The proposed architecture is designed to address efficiency and output-representation limitations in existing audio transformers.
- Results: HTS-AT achieves or equals state-of-the-art results on AudioSet, ESC-50, and Speech Command V2.The reported benchmark results cover multiple audio-classification scenarios.
- Efficiency: 31M vs. 87M parameters and 80 hrs vs. 600 hrs of training time are reported for HTS-AT versus AST at best performance.The comparison also reports fewer GPU-memory requirements for HTS-AT.
- Event localization: The token-semantic module enables event start-and-end localization from weakly labeled data, with better performance than previous CNN-based models.This extends the audio transformer beyond clip-level classification toward temporal event detection.
2. PROPOSED MODEL
HTS-AT reduces transformer computation through hierarchical patch merging and window attention, then adds token-semantic mapping for class-specific temporal activation maps. These design choices support efficient classification and event localization.
- 2.1. Hierarchical Transformer with Window Attention: A hierarchical transformer and window attention reduce the memory and computation costs caused by long audio-token sequences.Window attention computes local attention matrices rather than one global matrix, while hierarchical processing progressively reduces sequence size.
- 2.1.1. Encode the Audio Spectrogram: Audio mel-spectrograms are patchified and ordered by time, frequency, and window so adjacent tokens capture frequency relationships within time frames.The ordering is tailored to audio spectrograms, whose time dimension is typically longer than the frequency dimension.
- 2.1.2. Patch-Merge and Window Attention: Patch-Merge layers reshape token sequences into 2D maps and merge adjacent patches between transformer groups.After four groups, the token shape is reduced by 8 times, reducing GPU-memory consumption exponentially after each group.
- 2.1.2. Patch-Merge and Window Attention: Window attention computes relations within nonoverlapping M × M windows instead of across all tokens globally.As Patch-Merge combines adjacent windows in deeper layers, attention covers progressively larger regions.
- 2.2. Token Semantic Module: The token-semantic CNN maps final transformer outputs into event-class featuremaps and averages them for classification loss.The resulting event presence map preserves temporal structure for localization while producing class scores for binary cross-entropy training.
3. EXPERIMENTS
Experiments evaluate HTS-AT across classification, keyword spotting, and localization datasets, showing strong benchmark performance alongside reduced model size and training time. The token-semantic module supports event localization, while hierarchical design improves efficiency and unpretrained scalability.
- Evaluation Setup: HTS-AT is evaluated on AudioSet, ESC-50, Speech Command V2, and DESED across classification, keyword spotting, and event detection tasks.
- AudioSet Results: 0.471 mAP is achieved by the best single HTS-AT on AudioSet, exceeding AST’s 0.459.An ensemble of six HTS-AT models reaches 0.487 mAP, compared with AST ensembles at 0.475 and 0.485.
- Ablation and Pretraining: 0.440 mAP is obtained by unpretrained HTS-AT, rising to 0.453 with the token-semantic module and 0.471 with pretraining.Unpretrained AST reaches 0.366, 9.3% below its pretrained 0.459 result.
- Transfer Results: 97.0% accuracy establishes a new SOTA on ESC-50, while 98.0% on Speech Command V2 equals the SOTA.HTS-AT also shows smaller deviations than AST on these datasets.
- Localization: 50.7% average event-based F1 on DESED exceeds PANN, with HTS-AT scoring higher on 8 of 10 classes.The token-semantic module produces localization outputs, although Speech and Cleaner have relatively low F1-scores.
4. CONCLUSION AND FUTURE WORK
HTS-AT is presented as a high-performance, scalable, lightweight audio transformer that achieves strong results across audio classification scenarios. Its token-semantic module also supports event start- and end-time localization, with further localization and downstream-task studies proposed as future work.
- HTS-AT achieves a new SOTA across multiple datasets and audio classification scenarios.
- The token-semantic module enables localization of event start and end times.
- Future work will examine detailed localization training with partially strongly labeled AudioSet data and integration into downstream tasks.