Source-linked AI summary
STFT Frontend Mismatch in Pretrained Audio Spectrogram Transformers: Analysis and Adaptation
Julia Wilkins, Wei-Ching Lin, Abinaya Kumar, Shabnam Ghaffarzadegan, Juan Pablo Bellow
TL;DR
Pretrained AST inherits a fixed STFT frontend even though downstream tasks and resource constraints can call for different resolutions, and retraining for one configuration does not ensure robustness to unseen ones. The paper studies this mismatch and uses FiLM conditioning to adapt pretrained AST to variable STFT settings with limited additional training. FiLM improves classification across unseen frontend settings while maintaining strong performance in the original configuration.
Problem
Pretrained AST typically inherits a fixed STFT configuration, although downstream tasks and resource constraints may call for different resolutions; retraining for one configuration does not ensure robustness to unseen ones.
Source: §1 · p. 1Method
The study tests AST sensitivity to window and hop changes and uses FiLM conditioning to adapt pretrained AST to variable settings with limited additional training.
Source: §1 · p. 1, §3.3 · p. 2Results
FiLM adaptation improves audio classification across unseen frontend settings while maintaining strong performance in the original configuration, with substantial gains achievable using as few as 1, 000 additional training samples.
Source: §1 · p. 1Takeaways & Limitations
Within the studied AST classification setting, FiLM offers an adaptation route using limited additional data.
Source: §1 · p. 1The study varies only STFT window and hop length, keeping sample rate, Mel-bin count, and window type fixed.
Source: §3.2 · p. 2
Abstract
from arXiv · showhide
Pretrained audio spectrogram transformers are widely used as backbones of machine listening systems for sound event classification. However, they are typically trained using spectrograms with a fixed Short-Time Fourier Transform (STFT) configuration, implicitly binding the model to the time-frequency resolution seen during pretraining. In practice, adapting these systems to new domains or computational constraints often requires different STFT frontend parameters. In this work, we conduct a controlled study of the sensitivity of the foundational Audio Spectrogram Transformer to variations in STFT window and hop size, showing that audio classification performance degrades substantially as frontend parameters deviate from pretraining settings. As a mitigation strategy, we introduce a lightweight conditioning mechanism using FiLM that enables a pretrained model to adapt to variable input configurations using limited additional training data. Our method generalizes well to unseen STFT configurations, improving model robustness without sacrificing performance in the original setting.
1. INTRODUCTION
Pretrained AST models inherit a fixed STFT configuration, although downstream tasks may require different time-frequency resolutions for acoustic or computational reasons. This study measures the resulting sensitivity and introduces FiLM conditioning to adapt AST across configurations with limited additional training.
1. INTRODUCTION
- Downstream tasks may require finer temporal resolution for transient sounds or coarser representations under latency or resource constraints, creating frontend mismatch with pretrained settings.The paper defines STFT frontend mismatch as a discrepancy between the time-frequency resolution assumed by the model and what a downstream dataset or task requires.Source: §1 · p. 1
- Cropping, padding, repeating, or interpolating spectrograms can distort time-frequency structure or discard information, while fine-tuning or retraining requires substantial data and compute.Retraining on another fixed configuration also does not ensure robustness to unseen resolutions.Source: §1 · p. 1
- Prior flexibility approaches still require retraining or fine-tuning substantial parts of AST, and do not jointly investigate changes in both window and hop length.Other cited approaches focus on resolution selection during training or sampling-rate invariance rather than pretrained-model adaptation to both STFT parameters.Source: §1 · p. 1
- A controlled study finds that AST classification performance degrades as STFT window and hop lengths depart from pretraining settings.The proposed FiLM conditioning adapts pretrained AST to variable STFT window and hop lengths with minimal additional training.Source: §1 · p. 1
- Substantial performance improvements are reported with as few as 1,000 additional training samples, while performance remains strong in the original configuration.The study uses original AST and AudioSet, varying window and hop length while controlling sample rate and other STFT hyperparameters.Source: §1 · p. 1
2. CHARACTERIZING STFT FRONTEND SENSITIVITY
Zero-shot evaluation on AudioSet shows that AST is sensitive to changes in STFT window and hop lengths. Performance declines substantially as those parameters diverge from the pretraining configuration.
2. CHARACTERIZING STFT FRONTEND SENSITIVITY
- Zero-shot evaluation compares AudioSet classification performance across STFT window and hop lengths with performance under the pretraining settings.The pretrained AST is frozen; the original window and hop lengths serve as the upper-bound reference.Source: §2 · p. 1
- 0.456 mAP at the original configuration falls by over 50% relatively to 0.224 when the window shrinks by 12.5 ms at the original window-hop ratio.For extreme shifts, including a 100-sample window or hop equal to window size, performance often falls below 0.1.Source: §2 · p. 2, Figure 1 · p. 2
- The sensitivity analysis quantifies degradation across window and hop variation and establishes baselines for evaluating adaptation methods.The authors describe this effect as not previously explicitly quantified for pretrained AST across these two STFT parameters.Source: §2 · p. 2
3. STFT FRONTEND ADAPTATION
The adaptation framework conditions a frozen AST on STFT window and hop lengths using FiLM. Training exposes the model to multiple configurations and evaluates generalization to held-out window lengths.
3.1. Audio Spectrogram Transformer
- AST converts log-Mel spectrograms into overlapping patch embeddings, adds class and distillation tokens and positional encoding, then processes the sequence with a Transformer encoder.The encoder outputs are averaged and passed to a linear classifier; AST is pretrained on AudioSet-unbalanced.Source: §3.1 · p. 2
3.2. Multi-scale Spectrogram Configuration Training
- Training varies window lengths and proportional hop lengths while holding sample rate, Mel bins, and window type fixed.The tested windows are 100, 200, 800, 1600, and 3200 samples; the original pretraining configuration is excluded so evaluation uses unseen parameters.Source: §3.2 · p. 2
- Five-fold evaluation holds out each window length once, and each training batch randomly samples a training window and a corresponding hop length.When FiLM is applied within Transformer layers, one parameter set is shared across layers; applying it before the feed-forward layer was most effective among tested locations.Source: §3.2 · p. 2, §3.3 · p. 3
3.3. Conditioning with FiLM
- FiLM adapts pretrained, frozen AST by applying learned feature-wise affine modulation to intermediate representations, using window and hop lengths as conditioning information.The conditioning vector is z = [log(N), log(h)]; the study freezes about 87 million AST parameters and trains 198,528 FiLM parameters.Source: §3.3 · pp. 2–3
4. EXPERIMENTAL DESIGN
The experiments use AudioSet and a frozen pretrained AST baseline to test frontend changes, with positional-encoding adjustment and lightweight training and evaluation procedures. They compare FiLM adaptation with limited fine-tuning baselines.
4.1. Dataset
- AudioSet-balanced training uses 19,644 clips across 527 classes, and the 17,979-clip evaluation set keeps the study in AST’s pretraining domain.The balanced subset was selected to illustrate adaptation with minimal additional training resources.Source: §4.1 · p. 3
4.2. Model Adaptation Considerations
- The method interpolates or crops pretrained positional encodings to match incoming patch sequences; this inference-time scaling is the only modified AST component and is not trainable.The adjustment avoids zero-padding or cropping spectrograms, which could lose information when smaller hops produce denser temporal representations.Source: §4.2 · p. 3
4.3. Training Details
- Models train for 30 epochs with Adam and binary cross-entropy, while few-shot adaptation uses subsets from 10 to 10,000 samples and up to 100 epochs.The experiments omit additional AST data augmentation to isolate the adaptation effect.Source: §4.3 · p. 3
4.4. Evaluation Metrics
- Evaluation tests 20 window-hop configurations per architecture across five folds and reports AudioSet mean average precision.Each fold holds out one window size and tests four associated window-hop ratios.Source: §4.4 · p. 3
4.5. Baseline Methods
- The frozen pretrained AST evaluated at its original STFT setting provides the upper-bound, while zero-shot mismatch tests and partial fine-tuning of the classifier head or positional encoding provide baselines.The study excludes full fine-tuning and training from scratch to focus on computationally efficient adaptation strategies.Source: §4.5 · p. 3
5. RESULTS
FiLM conditioning improves AST classification across unseen STFT configurations and preserves performance with the original frontend. The study also reports that useful adaptation gains require relatively few training samples.
5.1. Adapting AST to unseen configurations with FiLM
- Fine-tuning sometimes improves larger-window results but slightly hurts near the pretrained window; the smallest (N/5) and largest (N) hops consistently perform worst.The authors report no significant difference between fine-tuning the classifier head and positional encoding.Source: §5.1 · pp. 3–4
- FiLM-Encoder nearly doubles zero-shot mAP at N=1600 (0.166→0.313, +14.7pp), with gains at N=200 and N=800 and across interpolated and extrapolated window lengths.Figure 2 compares mAP across unseen window-hop configurations and degradation relative to the original AST upper-bound.Source: Figure 2 · p. 4, §5.1 · p. 4
5.2. Preserving performance in the original setting
- FiLM variants preserve original-setting performance, with FiLM-Encoder averaging 0.443 mAP, only −0.013 from the upper-bound; fine-tuning baselines suffer more there.The comparison evaluates each model using the original frontend parameters.Source: §5.2 · p. 4
5.3. How much data do you need for FiLM adaptation?
- With 1,000 samples, FiLM-Encoder achieves substantial gains over zero-shot; 5,000 samples approach full-dataset training, while 100 samples yield +5.6pp at N=1600.Figure 3 plots few-shot adaptation against sample count and marks full-dataset training at 20k samples.Source: Figure 3 · p. 4, §5.3 · p. 4
6. CONCLUSION
The study analyzes STFT frontend mismatch in the original AST setting and proposes FiLM conditioning. It reports substantial degradation with changed window and hop lengths, with much of the loss recovered using as few as 1,000 training samples while preserving original-setting performance.
6. CONCLUSION
- Audio classification degrades substantially as window and hop lengths deviate from pretraining settings; FiLM recovers much of the loss and preserves original-setting performance with as few as 1,000 training samples.Source: §6 · p. 4
- The study proposes a lightweight FiLM-based conditioning strategy to mitigate STFT frontend mismatch.Source: §6 · p. 4
- The controlled analysis is limited to the original AST model and pretraining dataset, varying STFT window and hop length.Source: §6 · p. 4