PaperlayerView exact revision on arXiv

Source-linked AI summary

Muting Tokens: Exploring Token Sparsification for Efficient Audio Transformers

Shreya Sajid, Fabrizio Julian Piva, Hiram Rayo Torres Rodriguez, Elif Ceren Gok Yildirim, Murat Onur Yildirim, Joaquin Vanschoren

arXiv:2610.31235v1cs.SDcs.LGeess.AS

TL;DR

Token sparsification is underexplored in audio, where longer inputs increase the number of tokens and the cost of self-attention. This paper evaluates pruning and merging across models, datasets, and training settings, and introduces frequency-aware strategies. Its reported results favor frequency-aware approaches over attention-only methods, while intensity-guided pruning is effective for longer inputs.

  • Problem

    Token sparsification remains underexplored in audio, despite longer spectrogram inputs yielding more tokens and making self-attention increasingly expensive.

    Source: §1 · pp. 1–2

  • Method

    The paper evaluates pruning and merging methods across audio datasets, model sizes, and training settings, including sparsification strategies that combine attention with frequency cues.

    Source: §1 · p. 2, §3.1 · p. 3

  • Results

    Frequency-aware methods, particularly Spectral Flatness and Low-Frequency Energy, consistently outperform attention-only methods, while intensity-based pruning with attention can outperform pure attention for inputs longer than 2 seconds.

    Source: §1 · p. 2, §6 · p. 9

  • Takeaways & Limitations

    The findings support incorporating audio-specific frequency and input cues into sparsification strategies rather than relying only on generic attention signals.

    Source: §6 · p. 9

Abstract

from arXiv · show

Transformer-based models have shown strong performance across diverse tasks, but their reliance on self-attention mechanisms often leads to high computational costs. Token sparsification, which reduces the number of tokens processed in transformer layers, offers a promising solution to maintain performance while improving efficiency. However, although widely studied in the vision and language domains, token sparsification remains underexplored in audio, with existing work often limited to a single method, model size, or dataset. This work presents a comprehensive evaluation of token sparsification techniques for audio transformers. We assess a range of token pruning and merging strategies across multiple datasets and model sizes, under both post-training and fine-tuning settings. Furthermore, we introduce a novel sparsification approach that combines attention scores with spectral cues derived from the frequency content of token embeddings. Our experiments show that frequency-aware approaches, particularly those based on spectral flatness and low-frequency energy, consistently outperform attention-only methods. We also find that sparsification based on spectrogram patch intensity is especially effective for longer audio inputs. These results highlight the value of incorporating frequency-domain and input-specific information into sparsification strategies to enhance performance in audio tasks.

1 INTRODUCTION

Token sparsification may reduce the quadratic cost of audio transformers, but its use in audio remains underexplored as longer spectrograms yield more tokens. This work evaluates pruning and merging across models, datasets, and training settings, and introduces frequency-aware and intensity-guided approaches.

1 INTRODUCTION

  • Token sparsification is underexplored in audio, where longer spectrogram inputs produce more tokens and can make self-attention increasingly expensive.Self-attention complexity scales quadratically with the number of input tokens.Source: §1 · pp. 1–2
  • Prior audio sparsification evidence was limited to two methods, one model size, and fine-tuning, leaving post-training settings and input duration underexplored.The cited study also did not investigate how audio duration may influence intensity pruning.Source: §1 · p. 2
  • The study systematically evaluates token pruning and merging across multiple model sizes and datasets, under both fine-tuning and post-training sparsification.Experiments use a unified training and inference pipeline.Source: §1 · p. 2
  • Frequency-aware methods, particularly Spectral Flatness and Low-Frequency Energy, consistently outperform attention-only methods across the reported experimental settings.The introduction also reports that spectral-flatness pruning improves performance across a wide range of model sizes and datasets.Source: §1 · p. 2
  • Intensity-based pruning combined with attention scores can outperform pure attention methods for audio inputs longer than 2 seconds.The reported finding applies across the authors’ experimental setup.Source: §1 · p. 2

2 RELATED WORK

Token sparsification reduces transformer token counts through pruning or merging, but audio evaluations have been limited in method, model, dataset, or training setting. The paper positions its broader assessment against these gaps.

2.1 Token Sparsification: Overview

  • Pruning removes tokens and risks discarding useful information, whereas merging combines tokens but may introduce noise through averaging.The study includes representative approaches from both categories to address the lack of systematic assessment in audio tasks.Source: §2.1 · p. 2

2.2 Token Sparsification Methods for Audio

  • Prior audio use of ToMe reported a 0.4 percentage-point mAP drop on AudioSet-2M, but only in one dataset and an MAE framework.Because MAE is designed for sparse inputs, its sensitivity to sparsification may be reduced, leaving performance on non-MAE models uncertain.Source: §2.2 · pp. 2–3
  • PaSST patchout drops spectrogram patches during training, while post-training sparsification avoids requiring training with patchout.The cited discussion presents post-training sparsification as a cost-efficient alternative when retraining is costly or infeasible.Source: §2.2 · p. 3

2.3 Studies on Token Sparsification for Audio

  • A recent audio study examined Top-K and intensity pruning only on ViT-Base and during fine-tuning, motivating broader comparisons across methods, sizes, and settings.The paper identifies this narrow scope as a gap its study addresses.Source: §2.3 · p. 3

3 METHODOLOGY

The methodology pairs established pruning and merging baselines with lightweight audio-specific token-importance heuristics. It evaluates these approaches across benchmarks in both post-training and sparsified fine-tuning settings.

3 METHODOLOGY

  • The evaluation covers established pruning and merging methods alongside lightweight importance strategies based on spectrogram intensity and frequency-domain features.The paper frames these as two complementary directions: systematic comparison of existing methods and audio-relevant token scoring.Source: §3 · p. 3

3.1 Token Sparsification Methods

  • Top-K ranks tokens by CLS attention, EViT preserves discarded-token information in an attention-weighted fused token, and ToMe averages the most similar matched pairs.The compared methods represent pruning and merging paradigms.Source: §3.1 · p. 3

3.2 Task-specific Token Importance Estimation Strategies

  • Intensity-guided pruning splits patches by a dataset-specific threshold, ranks high-intensity tokens by CLS attention, and fills any remaining Top-K slots from the low-intensity pool.Intensity is used as a coarse filter because high-energy patches can also reflect background noise or nondiscriminative events.Source: Figure 2 · p. 3, §3.2 · p. 4
  • Low-Frequency Energy scores each projected token embedding’s low-frequency content and combines that score with attention to select tokens.A Gaussian low-pass filter highlights low-frequency structure in the embeddings.Source: §3.2 · p. 4, Figure 3 · p. 4
  • Spectral Flatness is computed per token embedding and combined with CLS attention, with lower flatness treated as a proxy for more structured, content-rich tokens.The paper illustrates the distinction with token embeddings whose spectra have tonal peaks versus flatter, noise-like profiles.Source: §3.2 · p. 4, Figure 4 · p. 4, Figure 5 · p. 5

3.3 Evaluation Strategy

  • Methods are evaluated both during inference on already fine-tuned models and during fine-tuning from ImageNet-pretrained weights on the target audio dataset.The evaluation is intended to compare practical and performance implications, especially at lower keep rates.Source: §3.3 · p. 4

4 EXPERIMENTAL SETUP

The experiments evaluate token sparsification on three audio datasets using AST models of multiple sizes. They compare post-training and fine-tuning settings with keep rates applied uniformly across sparsified layers.

4 EXPERIMENTAL SETUP

  • The evaluation covers SPC, ESC50, and AS-20K, spanning 1-, 5-, and 10-second audio and different task scales.SPC contains 35 spoken-command classes, ESC50 has 50 environmental-sound classes, and AS-20K has 527 categories.Source: §4 · p. 5
  • All experiments use ImageNet-pretrained AST Base, Small, or Tiny backbones with 16 × 16 patches and overlap of 6 in time and frequency.Sparsified fine-tuning uses AST-B only; post-training sparsification evaluates all three model sizes.Source: §4 · p. 5
  • Sparsification is applied at three equally spaced transformer points, with keep rates of 0.25, 0.50, 0.70, or 0.90 uniformly across sparsified layers.The module is placed between each block’s attention and MLP branches.Source: §4 · p. 5
  • Training follows the original AST setup: AS-20K and ESC50 are fine-tuned for 25 epochs, while SPC is trained for 30.When sparsification is enabled, default masking strategies such as SpecAugment are disabled; compute is measured as per-instance FLOPs at batch size 1.Source: §4 · p. 5

5 RESULTS & ANALYSIS

Across post-training sparsification, frequency-aware methods perform strongly across model sizes and datasets, while fine-tuning reduces sensitivity to the chosen heuristic. Controlled studies find that SF consistently outperforms attention-only variants and that intensity-based pruning is more effective on longer inputs; the paper’s practical guidance reflects these conditions and tuning trade-offs.

5.1 Post-Training Sparsification

  • Frequency-based LFE and SF frequently rank among the top two methods across model sizes and datasets, and SF achieves the best performance across most AST-B keep rates and datasets.SF and LFE yield the highest performance across datasets despite slightly higher computational overhead than Top-K; lower keep rates reduce FLOPs.Source: §5.1 · pp. 5–6, Table 1 · p. 6, Table 2 · p. 7

5.2 Sparsified Fine-tuning

  • With sparsified fine-tuning, most methods maintain accuracy close to baseline at a 25% keep rate, and performance differences are less pronounced than after post-training sparsification.The authors attribute this reduced heuristic sensitivity to models adapting to the tokens retained during training; Figure 6 reports post-training AST-B results on SPC and ESC50 at a 50% keep rate, where SF and LFE outperform other methods in accuracy.Source: §5.2 · p. 6, Figure 6 · p. 6

5.3 Comparative & Controlled Studies

  • SF consistently outperforms its TopK-only counterparts across keep rates and attention variants, whereas LFE can underperform on AS-20K and ESC50.Overall, frequency-domain information yields modest but consistent gains over attention-only pruning in post-training settings.Source: §5.3 · pp. 6–7, Figure 8 · p. 8
  • Intensity-based pruning outperforms Top-K for audio segments longer than 2 seconds across AS-20K and ESC50, but can remove critical information from short clips.The authors associate the difference with greater redundancy and clearer low-intensity regions in longer recordings, versus little redundancy in short clips.Source: §5.3 · pp. 7–8

5.4 Practical Guidelines

  • SF and LFE are practical frequency-aware choices, but LFE requires dataset-specific low-pass cutoff calibration while SF requires no additional tuning.The guidance also says intensity pruning can outperform standard methods mainly for larger models on inputs longer than 2 seconds, but requires threshold tuning.Source: §5.4 · p. 8
  • ToMe can perform well on small models or specific tasks, but its effectiveness is less consistent across some tasks.The post-training results describe ToMe as competitive on ESC50 with AST-S and AST-T, but inconsistent on AST-B.Source: §5.1 · p. 5, §5.3 · p. 7, §5.4 · p. 9
  • During fine-tuning, even simple sparsification heuristics can perform well because the model adapts to sparsity.This is consistent with the finding that method differences are less pronounced in sparsified fine-tuning than in post-training sparsification.Source: §5.2 · p. 6, §5.4 · p. 9

6 CONCLUSION

The paper evaluates pruning and merging methods across audio-transformer model sizes, datasets, and deployment settings. It finds frequency-aware approaches effective, particularly in post-training sparsification, and concludes that audio-specific cues merit inclusion in sparsification strategies.

6 CONCLUSION

  • The study evaluates token pruning and merging across multiple model sizes, datasets, and deployment scenarios.Its results show frequency-aware methods, particularly those using spectral flatness and low-frequency energy, consistently outperform attention-only methods, especially in post-training sparsification.Source: §6 · p. 9
  • The authors introduce a strategy combining attention with spectral priors that is effective across a wide range of configurations.The conclusion identifies task-specific token estimation and real-time applications as future directions.Source: §6 · p. 9

Related papers

Loading 2610.31235v1…